Jul 16 12:45:11.359269 ip-10-0-139-160 systemd[1]: Starting Kubernetes Kubelet... Jul 16 12:45:14.763334 ip-10-0-139-160 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:45:14.763334 ip-10-0-139-160 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 16 12:45:14.763334 ip-10-0-139-160 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:45:14.763334 ip-10-0-139-160 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 16 12:45:14.763334 ip-10-0-139-160 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 16 12:45:14.764432 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.764120 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 16 12:45:14.768930 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768916 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:45:14.768930 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768930 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768934 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768937 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768942 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768946 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768950 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768952 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768955 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768958 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768961 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768964 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768966 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768970 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768972 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768975 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768978 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768980 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768983 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768986 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:45:14.769001 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768988 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768991 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768994 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768996 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.768999 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769001 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769004 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769007 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769010 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769012 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769015 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769018 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769022 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769025 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769028 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769031 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769034 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769037 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769040 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:45:14.769481 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769042 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769045 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769047 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769050 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769052 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769055 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769057 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769060 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769063 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769065 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769068 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769070 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769073 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769076 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769078 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769083 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769085 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769088 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769090 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769093 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:45:14.770204 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769095 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769098 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769100 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769103 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769105 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769108 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769110 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769113 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769115 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769118 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769120 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769123 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769125 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769127 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769130 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769133 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769135 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769138 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769140 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769143 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:45:14.770680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769145 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769148 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769150 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769153 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769155 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769158 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769161 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769164 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769167 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769170 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769174 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769178 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769180 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769183 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769185 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769188 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769191 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769193 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769195 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:45:14.771217 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.769198 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771302 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771317 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771321 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771324 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771327 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771330 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771333 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771336 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771339 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771341 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771344 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771347 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771349 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771352 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771354 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771357 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771359 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771362 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771365 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:45:14.771654 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771368 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771371 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771373 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771378 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771381 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771384 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771387 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771390 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771393 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771396 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771399 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771401 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771408 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771412 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771415 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771418 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771421 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771423 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771426 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:45:14.772138 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771429 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771432 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771435 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771438 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771440 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771443 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771445 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771448 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771451 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771454 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771456 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771459 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771462 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771464 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771466 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771469 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771471 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771474 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771476 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:45:14.772575 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771479 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771481 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771484 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771486 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771489 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771491 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771494 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771498 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771502 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771505 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771508 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771510 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771513 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771515 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771518 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771520 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771523 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771525 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771527 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771530 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:45:14.773033 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771533 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771535 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771538 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771540 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771543 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771546 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771549 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771552 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771555 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771558 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771563 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771566 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771568 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771571 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771574 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771577 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771580 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771583 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771585 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:45:14.773502 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771588 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771592 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.771595 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772011 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772017 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772021 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772025 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772029 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772032 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772034 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772037 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772039 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772042 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772045 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772047 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772050 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772053 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772055 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772057 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772060 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:45:14.773975 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772063 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772066 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772068 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772071 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772074 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772076 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772079 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772082 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772084 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772089 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772092 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772094 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772097 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772100 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772103 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772105 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772108 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772110 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772113 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:45:14.774456 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772116 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772118 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772121 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772124 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772128 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772132 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772135 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772138 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772140 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772143 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772146 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772149 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772152 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772155 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772160 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772163 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772166 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772169 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:45:14.774909 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772172 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772175 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772177 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772180 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772183 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772186 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772188 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772191 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772193 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772196 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772198 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772201 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772203 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772207 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772210 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772212 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772215 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772217 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772220 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772222 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:45:14.775356 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772225 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772227 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772229 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772232 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772234 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772237 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772240 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772243 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772246 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772248 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772251 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772253 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772256 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772258 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772260 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772263 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772266 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772269 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772271 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772273 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:45:14.775810 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772276 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772279 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772281 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772284 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772286 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772353 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772360 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772367 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772372 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772376 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772379 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772383 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772387 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772390 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772393 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772397 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772400 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772403 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772406 2576 flags.go:64] FLAG: --cgroup-root="" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772409 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772412 2576 flags.go:64] FLAG: --client-ca-file="" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772416 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772419 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 16 12:45:14.776315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772423 2576 flags.go:64] FLAG: --cluster-domain="" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772425 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772428 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772431 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772435 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772439 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772442 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772445 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772448 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772451 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772454 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772457 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772460 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772462 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772466 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772469 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772472 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772475 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772477 2576 flags.go:64] FLAG: --enable-server="true" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772480 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772485 2576 flags.go:64] FLAG: --event-burst="100" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772488 2576 flags.go:64] FLAG: --event-qps="50" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772491 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772493 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 16 12:45:14.776841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772496 2576 flags.go:64] FLAG: --eviction-hard="" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772500 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772503 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772506 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772509 2576 flags.go:64] FLAG: --eviction-soft="" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772512 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772514 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772519 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772522 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772525 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772528 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772530 2576 flags.go:64] FLAG: --feature-gates="" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772534 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772537 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772540 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772543 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772546 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772549 2576 flags.go:64] FLAG: --help="false" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772552 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-160.ec2.internal" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772555 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772557 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772560 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772564 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 16 12:45:14.777389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772567 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772570 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772572 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772575 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772578 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772581 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772584 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772587 2576 flags.go:64] FLAG: --kube-reserved="" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772590 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772592 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772597 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772600 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772603 2576 flags.go:64] FLAG: --lock-file="" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772605 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772608 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772611 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772617 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772620 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772623 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772626 2576 flags.go:64] FLAG: --logging-format="text" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772630 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772633 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772636 2576 flags.go:64] FLAG: --manifest-url="" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772639 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772643 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 16 12:45:14.777957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772646 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772650 2576 flags.go:64] FLAG: --max-pods="110" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772653 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772656 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772658 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772661 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772664 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772667 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772670 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772678 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772681 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772684 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772686 2576 flags.go:64] FLAG: --pod-cidr="" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772689 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772695 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772698 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772701 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772705 2576 flags.go:64] FLAG: --port="10250" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772708 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772710 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0398428f659cb4420" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772714 2576 flags.go:64] FLAG: --qos-reserved="" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772717 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772720 2576 flags.go:64] FLAG: --register-node="true" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772722 2576 flags.go:64] FLAG: --register-with-taints="" Jul 16 12:45:14.778549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772726 2576 flags.go:64] FLAG: --registry-burst="10" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772729 2576 flags.go:64] FLAG: --registry-qps="5" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772732 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772735 2576 flags.go:64] FLAG: --reserved-memory="" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772739 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772742 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772745 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772748 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772751 2576 flags.go:64] FLAG: --runonce="false" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772754 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772757 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772760 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772763 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772766 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772769 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772772 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772775 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772777 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772780 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772783 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772786 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772789 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772792 2576 flags.go:64] FLAG: --system-cgroups="" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772794 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772799 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 16 12:45:14.779129 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772803 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772806 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772810 2576 flags.go:64] FLAG: --tls-min-version="" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772812 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772828 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772831 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772834 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772837 2576 flags.go:64] FLAG: --v="2" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772841 2576 flags.go:64] FLAG: --version="false" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772855 2576 flags.go:64] FLAG: --vmodule="" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772859 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.772863 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772985 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772989 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772993 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772996 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.772998 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773001 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773004 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773008 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773010 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773013 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:45:14.779747 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773015 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773018 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773020 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773023 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773026 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773028 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773031 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773033 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773036 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773038 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773042 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773044 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773047 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773050 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773053 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773055 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773058 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773060 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773063 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773065 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:45:14.780270 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773068 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773073 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773075 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773078 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773081 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773083 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773086 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773088 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773091 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773094 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773096 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773099 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773101 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773104 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773107 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773110 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773113 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773115 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773118 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773120 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:45:14.780740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773123 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773125 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773128 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773131 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773134 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773136 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773139 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773141 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773144 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773146 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773149 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773151 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773154 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773159 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773162 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773164 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773166 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773169 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773171 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773174 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:45:14.781258 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773177 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773180 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773182 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773185 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773187 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773190 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773192 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773195 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773197 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773200 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773204 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773208 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773210 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773213 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773216 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773219 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773222 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773225 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773229 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:45:14.781722 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773232 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773235 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773238 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773241 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773246 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773248 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773252 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773255 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773258 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.773261 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.773904 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.780213 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.780228 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.781102 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 16 12:45:14.782193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.781113 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781171 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781177 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781180 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781183 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781187 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781190 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781193 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781197 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781200 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781203 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781206 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781208 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781211 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781214 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781216 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781219 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781221 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781224 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:45:14.782526 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781227 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781229 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781232 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781234 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781237 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781239 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781242 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781244 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781247 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781249 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781252 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781256 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781259 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781261 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781264 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781267 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781270 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781272 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781275 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781277 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:45:14.782990 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781280 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781283 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781285 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781288 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781291 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781293 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781296 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781299 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781303 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781306 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781309 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781312 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781314 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781317 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781320 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781323 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781325 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781328 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781330 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:45:14.783455 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781333 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781335 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781338 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781341 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781344 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781347 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781350 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781352 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781355 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781357 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781360 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781362 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781365 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781367 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781370 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781372 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781375 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781377 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781380 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781383 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:45:14.783923 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781386 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781388 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781391 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781394 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781397 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781399 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781402 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781404 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781407 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781410 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781414 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781417 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781420 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781422 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781425 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781428 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781430 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781432 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781435 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781438 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:45:14.784395 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781440 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781443 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.781448 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781555 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781559 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781562 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781565 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781567 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781570 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781573 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781576 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781579 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781581 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781584 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 16 12:45:14.784865 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781586 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781589 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781591 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781594 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781596 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781599 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781603 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781606 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781609 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781611 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781615 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781617 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781620 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781623 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781626 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781628 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781632 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781636 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781639 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 16 12:45:14.785227 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781642 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781644 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781647 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781650 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781652 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781655 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781658 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781660 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781664 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781666 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781669 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781671 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781674 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781676 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781679 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781681 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781684 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781686 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781689 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781691 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 16 12:45:14.785669 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781694 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781696 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781699 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781702 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781704 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781706 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781709 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781711 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781714 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781717 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781721 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781724 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781727 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781730 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781733 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781735 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781738 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781740 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781743 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781745 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 16 12:45:14.786290 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781748 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781750 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781753 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781755 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781758 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781761 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781763 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781766 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781768 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781771 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781773 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781775 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781778 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781780 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781782 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781785 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781788 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781790 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781793 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781795 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 16 12:45:14.786752 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781798 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781800 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781803 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781805 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781808 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781811 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781828 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781832 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:14.781834 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.781839 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.781932 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.783870 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.785113 2576 server.go:1019] "Starting client certificate rotation" Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.785214 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:45:14.787247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.786048 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 16 12:45:14.787829 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.787803 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:45:14.788929 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.788909 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 16 12:45:14.800147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.800125 2576 log.go:25] "Validated CRI v1 runtime API" Jul 16 12:45:14.801963 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.801948 2576 log.go:25] "Validated CRI v1 image API" Jul 16 12:45:14.805779 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.805759 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 16 12:45:14.808306 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.808278 2576 fs.go:135] Filesystem UUIDs: map[1aeb80d1-cd6e-4e9a-b305-3ead67556d32:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89ecb9c0-0c07-44ac-92ab-f359a03d5849:/dev/nvme0n1p4] Jul 16 12:45:14.808374 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.808304 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 16 12:45:14.814365 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.814252 2576 manager.go:216] Machine: {Timestamp:2026-07-16 12:45:14.812134967 +0000 UTC m=+0.354741706 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101059 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20264320d7a81282a97a860e7d54a7 SystemUUID:ec202643-20d7-a812-82a9-7a860e7d54a7 BootID:a9d58191-e0ca-46ec-8dc7-df682b66660e Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8a:a7:21:f8:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8a:a7:21:f8:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:13:9f:ea:e5:6a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 16 12:45:14.814365 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.814363 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 16 12:45:14.814478 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.814441 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 16 12:45:14.817671 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.817640 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 16 12:45:14.817834 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.817676 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 16 12:45:14.818354 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.818344 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 16 12:45:14.818385 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.818357 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 16 12:45:14.818385 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.818365 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:45:14.819094 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.819083 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 16 12:45:14.819967 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.819948 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:45:14.820375 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.820365 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 16 12:45:14.822903 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.822892 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:45:14.823016 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.823006 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 16 12:45:14.823047 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.823036 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 16 12:45:14.823047 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.823046 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 16 12:45:14.823105 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.823058 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 16 12:45:14.823105 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.823069 2576 kubelet.go:398] "Adding apiserver pod source" Jul 16 12:45:14.823105 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.823077 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 16 12:45:14.823459 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.823447 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:45:14.823506 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.823468 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 16 12:45:14.825472 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.825455 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 16 12:45:14.827200 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.827188 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 16 12:45:14.827241 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.827217 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 16 12:45:14.827369 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.827360 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 16 12:45:14.827411 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.827375 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 16 12:45:14.827411 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.827382 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 16 12:45:14.827411 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.827388 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 16 12:45:14.827411 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.827398 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 16 12:45:14.827411 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.827403 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 16 12:45:14.828329 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.828320 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 16 12:45:14.828361 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.828330 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 16 12:45:14.828361 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.828337 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 16 12:45:14.828361 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.828342 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 16 12:45:14.828361 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.828356 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 16 12:45:14.828465 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.828364 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 16 12:45:14.829200 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.829191 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 16 12:45:14.829200 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.829200 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 16 12:45:14.832340 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.832323 2576 server.go:1268] "Started kubelet" Jul 16 12:45:14.832488 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.832427 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 16 12:45:14.832571 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.832504 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 16 12:45:14.832571 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.832454 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 16 12:45:14.833331 ip-10-0-139-160 systemd[1]: Started Kubernetes Kubelet. Jul 16 12:45:14.833887 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.833739 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 16 12:45:14.834320 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.834241 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 16 12:45:14.834919 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.834906 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 16 12:45:14.838983 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.838940 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 16 12:45:14.840442 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.840420 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 16 12:45:14.840505 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.840440 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 16 12:45:14.841602 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.840716 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-160.ec2.internal.18c2c5d38fbaf157 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-160.ec2.internal,UID:ip-10-0-139-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-160.ec2.internal,},FirstTimestamp:2026-07-16 12:45:14.832302423 +0000 UTC m=+0.374909161,LastTimestamp:2026-07-16 12:45:14.832302423 +0000 UTC m=+0.374909161,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-160.ec2.internal,}" Jul 16 12:45:14.842587 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.842567 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 16 12:45:14.842669 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.842588 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 16 12:45:14.842836 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.842733 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:14.843386 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843126 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 16 12:45:14.843386 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843137 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 16 12:45:14.843386 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843149 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 16 12:45:14.843386 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843228 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 16 12:45:14.843386 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843240 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 16 12:45:14.843659 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843603 2576 factory.go:55] Registering systemd factory Jul 16 12:45:14.843659 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843617 2576 factory.go:223] Registration of the systemd container factory successfully Jul 16 12:45:14.843860 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843842 2576 factory.go:153] Registering CRI-O factory Jul 16 12:45:14.843947 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843864 2576 factory.go:223] Registration of the crio container factory successfully Jul 16 12:45:14.843947 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843917 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 16 12:45:14.843947 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843942 2576 factory.go:103] Registering Raw factory Jul 16 12:45:14.844070 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.843955 2576 manager.go:1196] Started watching for new ooms in manager Jul 16 12:45:14.844549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.844532 2576 manager.go:349] Starting recovery of all containers Jul 16 12:45:14.848876 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.848851 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 16 12:45:14.849070 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.849047 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 16 12:45:14.851043 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.850893 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 16 12:45:14.854358 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.854340 2576 manager.go:354] Recovery completed Jul 16 12:45:14.859320 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.859308 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:45:14.861077 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.861061 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:45:14.861158 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.861086 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:45:14.861158 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.861098 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:45:14.862030 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.862013 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 16 12:45:14.862030 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.862029 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 16 12:45:14.862125 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.862032 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:45:14.862125 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.862046 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 16 12:45:14.862798 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.862783 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:45:14.862859 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.862827 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:45:14.862859 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.862847 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:45:14.862935 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.862907 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ff5qz" Jul 16 12:45:14.863215 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.863095 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-160.ec2.internal.18c2c5d39171f5fb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-160.ec2.internal,UID:ip-10-0-139-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-160.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-160.ec2.internal,},FirstTimestamp:2026-07-16 12:45:14.861073915 +0000 UTC m=+0.403680653,LastTimestamp:2026-07-16 12:45:14.861073915 +0000 UTC m=+0.403680653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-160.ec2.internal,}" Jul 16 12:45:14.863280 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.863258 2576 policy_none.go:49] "None policy: Start" Jul 16 12:45:14.863280 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.863276 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 16 12:45:14.863361 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.863292 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 16 12:45:14.864292 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.864279 2576 policy_none.go:47] "Start" Jul 16 12:45:14.872802 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.872740 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-160.ec2.internal.18c2c5d3917239f3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-160.ec2.internal,UID:ip-10-0-139-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-160.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-160.ec2.internal,},FirstTimestamp:2026-07-16 12:45:14.861091315 +0000 UTC m=+0.403698052,LastTimestamp:2026-07-16 12:45:14.861091315 +0000 UTC m=+0.403698052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-160.ec2.internal,}" Jul 16 12:45:14.874062 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.874045 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ff5qz" Jul 16 12:45:14.897399 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.897384 2576 manager.go:337] "Starting Device Plugin manager" Jul 16 12:45:14.897465 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.897421 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 16 12:45:14.897465 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.897430 2576 server.go:87] "Starting device plugin registration server" Jul 16 12:45:14.897678 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.897664 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 16 12:45:14.897757 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.897679 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 16 12:45:14.897841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.897751 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 16 12:45:14.897895 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.897842 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 16 12:45:14.897895 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.897850 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 16 12:45:14.898494 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.898463 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 16 12:45:14.898579 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.898512 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:14.979843 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.979800 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 16 12:45:14.980809 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.979856 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 16 12:45:14.980809 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.979879 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 16 12:45:14.980809 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:14.979916 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 16 12:45:14.982378 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.982356 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:45:14.998408 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.998393 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:45:14.999426 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.999412 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:45:14.999487 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.999440 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:45:14.999487 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.999454 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:45:14.999487 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:14.999481 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.008949 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.008933 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.008996 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.008953 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-160.ec2.internal\": node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.055903 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.055830 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.080150 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.080118 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal"] Jul 16 12:45:15.080306 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.080201 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:45:15.081148 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.081131 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:45:15.081213 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.081161 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:45:15.081213 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.081173 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:45:15.082302 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.082290 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:45:15.082428 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.082414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.082467 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.082444 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:45:15.083033 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.083011 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:45:15.083107 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.083041 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:45:15.083107 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.083056 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:45:15.083107 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.083012 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:45:15.083196 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.083123 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:45:15.083196 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.083132 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:45:15.085113 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.085097 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.085165 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.085124 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:45:15.085778 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.085764 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:45:15.085869 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.085788 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:45:15.085869 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.085799 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:45:15.111522 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.111501 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-160.ec2.internal\" not found" node="ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.115849 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.115833 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-160.ec2.internal\" not found" node="ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.145322 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.145302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02c19a6ca1b613e10d3b866f18f227c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal\" (UID: \"02c19a6ca1b613e10d3b866f18f227c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.145400 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.145326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c19a6ca1b613e10d3b866f18f227c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal\" (UID: \"02c19a6ca1b613e10d3b866f18f227c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.145400 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.145342 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0bbc06f37afebb87bdba8d241d51d6dd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-160.ec2.internal\" (UID: \"0bbc06f37afebb87bdba8d241d51d6dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.156096 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.156078 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.246197 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.246165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02c19a6ca1b613e10d3b866f18f227c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal\" (UID: \"02c19a6ca1b613e10d3b866f18f227c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.246197 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.246200 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c19a6ca1b613e10d3b866f18f227c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal\" (UID: \"02c19a6ca1b613e10d3b866f18f227c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.246343 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.246216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0bbc06f37afebb87bdba8d241d51d6dd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-160.ec2.internal\" (UID: \"0bbc06f37afebb87bdba8d241d51d6dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.246343 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.246266 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02c19a6ca1b613e10d3b866f18f227c8-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal\" (UID: \"02c19a6ca1b613e10d3b866f18f227c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.246343 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.246272 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/02c19a6ca1b613e10d3b866f18f227c8-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal\" (UID: \"02c19a6ca1b613e10d3b866f18f227c8\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.246343 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.246315 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0bbc06f37afebb87bdba8d241d51d6dd-config\") pod \"kube-apiserver-proxy-ip-10-0-139-160.ec2.internal\" (UID: \"0bbc06f37afebb87bdba8d241d51d6dd\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.256406 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.256376 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.356795 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.356731 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.413948 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.413923 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.418673 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.418656 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" Jul 16 12:45:15.457635 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.457609 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.558098 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.558075 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.658618 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.658545 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.759080 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.759050 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.784183 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.784155 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 16 12:45:15.784733 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.784281 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 16 12:45:15.849330 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.849308 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:45:15.860085 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.860060 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.876691 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.876657 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 12:40:14 +0000 UTC" deadline="2027-12-23 03:58:22.688426509 +0000 UTC" Jul 16 12:45:15.876691 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.876687 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12591h13m6.811742838s" Jul 16 12:45:15.920042 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.919978 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 16 12:45:15.921205 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.921172 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientMemory" Jul 16 12:45:15.921312 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.921210 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasNoDiskPressure" Jul 16 12:45:15.921312 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.921226 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeHasSufficientPID" Jul 16 12:45:15.941731 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:15.941706 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbc06f37afebb87bdba8d241d51d6dd.slice/crio-c1d3b325547c8690c8757bc70f6ec65fd44a1f5fc258f97d65975f99dd9b4712 WatchSource:0}: Error finding container c1d3b325547c8690c8757bc70f6ec65fd44a1f5fc258f97d65975f99dd9b4712: Status 404 returned error can't find the container with id c1d3b325547c8690c8757bc70f6ec65fd44a1f5fc258f97d65975f99dd9b4712 Jul 16 12:45:15.941906 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:15.941885 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c19a6ca1b613e10d3b866f18f227c8.slice/crio-ad846fa4ce768240114c0734326143ae1f2f887b7bc04d510b0a965fc9a7d8d5 WatchSource:0}: Error finding container ad846fa4ce768240114c0734326143ae1f2f887b7bc04d510b0a965fc9a7d8d5: Status 404 returned error can't find the container with id ad846fa4ce768240114c0734326143ae1f2f887b7bc04d510b0a965fc9a7d8d5 Jul 16 12:45:15.949456 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.949438 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:45:15.960757 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:15.960737 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:15.983046 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.983002 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" event={"ID":"02c19a6ca1b613e10d3b866f18f227c8","Type":"ContainerStarted","Data":"ad846fa4ce768240114c0734326143ae1f2f887b7bc04d510b0a965fc9a7d8d5"} Jul 16 12:45:15.983994 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:15.983973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" event={"ID":"0bbc06f37afebb87bdba8d241d51d6dd","Type":"ContainerStarted","Data":"c1d3b325547c8690c8757bc70f6ec65fd44a1f5fc258f97d65975f99dd9b4712"} Jul 16 12:45:16.061497 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.061471 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:16.161935 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.161911 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:16.262504 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.262433 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:16.318393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.318369 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:45:16.362958 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.362936 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-160.ec2.internal\" not found" Jul 16 12:45:16.391706 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.391685 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:45:16.443055 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.443030 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" Jul 16 12:45:16.458300 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.458268 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:45:16.459415 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.459393 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" Jul 16 12:45:16.467241 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.467150 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 16 12:45:16.823650 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.823620 2576 apiserver.go:52] "Watching apiserver" Jul 16 12:45:16.835639 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.835604 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 16 12:45:16.836726 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.836695 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-d62t6","kube-system/konnectivity-agent-gzltq","kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf","openshift-cluster-node-tuning-operator/tuned-g4jcs","openshift-dns/node-resolver-l764j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal","openshift-multus/network-metrics-daemon-hpwb2","openshift-network-operator/iptables-alerter-wb2vs","kube-system/global-pull-secret-syncer-pvcrz","openshift-image-registry/node-ca-ztrvw","openshift-multus/multus-additional-cni-plugins-2kn5j","openshift-multus/multus-sq748","openshift-network-diagnostics/network-check-target-kmxqh"] Jul 16 12:45:16.838352 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.838328 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.838505 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.838473 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:16.839412 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.839383 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sq748" Jul 16 12:45:16.840502 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.840438 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:16.840650 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.840513 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:16.841791 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.841565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.842482 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.842460 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 16 12:45:16.842880 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.842758 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 16 12:45:16.842979 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.842920 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k2vpk\"" Jul 16 12:45:16.842979 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.842960 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 16 12:45:16.843107 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.843078 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.844060 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.843950 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 16 12:45:16.844482 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.844268 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 16 12:45:16.844482 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.844290 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t65lt\"" Jul 16 12:45:16.844482 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.844345 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 16 12:45:16.844482 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.844415 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 16 12:45:16.844707 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.844272 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 16 12:45:16.844707 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.844577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.845525 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.845506 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:16.846180 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.846074 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-nq4gr\"" Jul 16 12:45:16.846443 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.846423 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:45:16.846553 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.846461 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 16 12:45:16.847350 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.847329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:16.847490 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.847465 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:16.847752 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.847602 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 16 12:45:16.847752 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.847628 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 16 12:45:16.848036 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.848016 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:45:16.848266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.848229 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4h5t\"" Jul 16 12:45:16.848355 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.848327 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rgqqm\"" Jul 16 12:45:16.848869 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.848494 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:16.848988 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.848969 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 16 12:45:16.849053 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.848976 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 16 12:45:16.851238 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.850716 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.851238 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.850787 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 16 12:45:16.851401 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.851286 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 16 12:45:16.851911 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.851894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:16.851998 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.851970 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w4fcj\"" Jul 16 12:45:16.853451 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.853420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.853861 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.853844 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 16 12:45:16.854148 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854128 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 16 12:45:16.854225 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-socket-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.854225 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-registration-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.854225 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysconfig\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854225 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854221 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-lib-modules\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854415 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854269 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c2a4180-949b-4c61-897c-9c8dbb79e831-tmp\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854415 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-daemon-config\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.854415 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854329 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysctl-conf\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854415 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854351 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-cni-multus\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.854415 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854374 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-kubelet\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.854415 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854397 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-hostroot\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-var-lib-kubelet\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854447 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-host\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4642c698-57ae-49ad-a5f5-54c575f0fde9-tmp-dir\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-cnibin\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-socket-dir-parent\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bhd8\" (UniqueName: \"kubernetes.io/projected/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-kube-api-access-5bhd8\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-systemd\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-run\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69tv\" (UniqueName: \"kubernetes.io/projected/3c2a4180-949b-4c61-897c-9c8dbb79e831-kube-api-access-v69tv\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.854654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-k8s-cni-cncf-io\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854696 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-device-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-sys-fs\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4642c698-57ae-49ad-a5f5-54c575f0fde9-hosts-file\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6n7\" (UniqueName: \"kubernetes.io/projected/4642c698-57ae-49ad-a5f5-54c575f0fde9-kube-api-access-7l6n7\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854802 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b6659fe-fabb-4a30-aaba-3615b0e73f11-kubelet-config\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b6659fe-fabb-4a30-aaba-3615b0e73f11-dbus\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-cni-binary-copy\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-conf-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.854998 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855018 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2pc9z\"" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-system-cni-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855098 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-os-release\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855128 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855134 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-multus-certs\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855158 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-etc-kubernetes\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnh4d\" (UniqueName: \"kubernetes.io/projected/fd0fb912-647a-444a-872f-3d18d1990d56-kube-api-access-cnh4d\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855234 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-modprobe-d\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855317 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-kubernetes\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855362 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysctl-d\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855387 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855425 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tztl2\"" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855469 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-netns\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855470 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855407 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-cni-bin\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-sys\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855714 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-tuned\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-cni-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.855810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.855757 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 16 12:45:16.856670 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.856068 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mkljl\"" Jul 16 12:45:16.856670 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.856262 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 16 12:45:16.856670 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.856324 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 16 12:45:16.860374 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.860355 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 16 12:45:16.882057 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.882032 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8vbz2" Jul 16 12:45:16.893405 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.893365 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8vbz2" Jul 16 12:45:16.944623 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.944596 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 16 12:45:16.956409 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956375 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/743b5ace-80ef-4034-a39e-799daffaab00-iptables-alerter-script\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:16.956555 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956416 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-run-netns\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.956555 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-env-overrides\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.956555 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956468 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-kubelet\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.956555 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956511 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-host\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.956555 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-cni-netd\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.956851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956577 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-kubelet\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.956851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-os-release\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.956851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-host\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.956851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-socket-dir-parent\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.956851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bhd8\" (UniqueName: \"kubernetes.io/projected/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-kube-api-access-5bhd8\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.956851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-systemd\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.956851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-run\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956856 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-socket-dir-parent\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956889 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-systemd\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/743b5ace-80ef-4034-a39e-799daffaab00-host-slash\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-run\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.956977 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-k8s-cni-cncf-io\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc0dd1b8-c7e2-44a2-b653-135b66d1654b-agent-certs\") pod \"konnectivity-agent-gzltq\" (UID: \"fc0dd1b8-c7e2-44a2-b653-135b66d1654b\") " pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957023 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b6659fe-fabb-4a30-aaba-3615b0e73f11-kubelet-config\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957051 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-cni-binary-copy\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957083 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-k8s-cni-cncf-io\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b6659fe-fabb-4a30-aaba-3615b0e73f11-kubelet-config\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.957147 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957119 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-conf-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957219 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957245 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957286 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-log-socket\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-system-cni-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957337 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-os-release\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957369 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-system-cni-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957386 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-multus-certs\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957404 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-etc-kubernetes\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnh4d\" (UniqueName: \"kubernetes.io/projected/fd0fb912-647a-444a-872f-3d18d1990d56-kube-api-access-cnh4d\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysctl-d\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-systemd\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957566 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysctl-d\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-cni-bin\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957602 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-var-lib-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.957651 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-ovn\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957666 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hw24\" (UniqueName: \"kubernetes.io/projected/caf2381b-6027-4b9d-bea9-f20120aec027-kube-api-access-2hw24\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957687 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-cni-binary-copy\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-multus-certs\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-cni-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957734 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-registration-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957741 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-conf-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957771 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-etc-selinux\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957783 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-registration-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957813 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-cni-bin\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957839 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-cni-dir\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-ovnkube-config\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957877 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-etc-kubernetes\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-daemon-config\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957949 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-os-release\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.957982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysctl-conf\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.958327 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958019 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcd6l\" (UniqueName: \"kubernetes.io/projected/5cf65300-74d2-4492-8245-b3984de67048-kube-api-access-kcd6l\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-run-ovn-kubernetes\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5664fccb-777b-4367-8239-458af04f6c29-host\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958141 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysctl-conf\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-cni-multus\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958174 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-hostroot\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958180 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-var-lib-cni-multus\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-var-lib-kubelet\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4642c698-57ae-49ad-a5f5-54c575f0fde9-tmp-dir\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958228 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-hostroot\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958237 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958263 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-ovnkube-script-lib\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958271 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-multus-daemon-config\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-var-lib-kubelet\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-system-cni-dir\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958312 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958338 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-cnibin\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.959072 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69tv\" (UniqueName: \"kubernetes.io/projected/3c2a4180-949b-4c61-897c-9c8dbb79e831-kube-api-access-v69tv\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958385 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkmq\" (UniqueName: \"kubernetes.io/projected/743b5ace-80ef-4034-a39e-799daffaab00-kube-api-access-gdkmq\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-kubelet\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-cnibin\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958473 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-etc-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958519 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4642c698-57ae-49ad-a5f5-54c575f0fde9-tmp-dir\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-device-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-sys-fs\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958627 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-device-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958630 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-sys-fs\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.958641 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958664 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4642c698-57ae-49ad-a5f5-54c575f0fde9-hosts-file\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958691 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6n7\" (UniqueName: \"kubernetes.io/projected/4642c698-57ae-49ad-a5f5-54c575f0fde9-kube-api-access-7l6n7\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.958726 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret podName:0b6659fe-fabb-4a30-aaba-3615b0e73f11 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:17.458696051 +0000 UTC m=+3.001302789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret") pod "global-pull-secret-syncer-pvcrz" (UID: "0b6659fe-fabb-4a30-aaba-3615b0e73f11") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zph46\" (UniqueName: \"kubernetes.io/projected/5664fccb-777b-4367-8239-458af04f6c29-kube-api-access-zph46\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:16.959786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958787 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4642c698-57ae-49ad-a5f5-54c575f0fde9-hosts-file\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958799 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b6659fe-fabb-4a30-aaba-3615b0e73f11-dbus\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc0dd1b8-c7e2-44a2-b653-135b66d1654b-konnectivity-ca\") pod \"konnectivity-agent-gzltq\" (UID: \"fc0dd1b8-c7e2-44a2-b653-135b66d1654b\") " pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958860 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-cni-bin\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958873 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-cnibin\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsc2z\" (UniqueName: \"kubernetes.io/projected/d2c635b3-06ef-492c-967e-f051ce3a388d-kube-api-access-bsc2z\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958956 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b6659fe-fabb-4a30-aaba-3615b0e73f11-dbus\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-modprobe-d\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.958990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-kubernetes\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959049 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-kubernetes\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959065 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959084 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-modprobe-d\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-netns\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-sys\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959168 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-host-run-netns\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.960294 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959189 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-tuned\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959229 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-systemd-units\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959244 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-sys\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-slash\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-node-log\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/caf2381b-6027-4b9d-bea9-f20120aec027-ovn-node-metrics-cert\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959339 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-socket-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959380 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysconfig\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-lib-modules\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c2a4180-949b-4c61-897c-9c8dbb79e831-tmp\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959464 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959488 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-sysconfig\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3c2a4180-949b-4c61-897c-9c8dbb79e831-lib-modules\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959536 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/fd0fb912-647a-444a-872f-3d18d1990d56-socket-dir\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5664fccb-777b-4367-8239-458af04f6c29-serviceca\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:16.960806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.959604 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:16.962335 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.962299 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c2a4180-949b-4c61-897c-9c8dbb79e831-tmp\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.962440 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.962340 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3c2a4180-949b-4c61-897c-9c8dbb79e831-etc-tuned\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.967923 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.967903 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:16.968026 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.967931 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:16.968026 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.967941 2576 projected.go:196] Error preparing data for projected volume kube-api-access-scjvm for pod openshift-network-diagnostics/network-check-target-kmxqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:16.968026 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:16.968011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm podName:ad46a259-f207-45c2-8379-5040cab7d186 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:17.467995809 +0000 UTC m=+3.010602555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-scjvm" (UniqueName: "kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm") pod "network-check-target-kmxqh" (UID: "ad46a259-f207-45c2-8379-5040cab7d186") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:16.970548 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.970522 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnh4d\" (UniqueName: \"kubernetes.io/projected/fd0fb912-647a-444a-872f-3d18d1990d56-kube-api-access-cnh4d\") pod \"aws-ebs-csi-driver-node-tzggf\" (UID: \"fd0fb912-647a-444a-872f-3d18d1990d56\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:16.970914 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.970895 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6n7\" (UniqueName: \"kubernetes.io/projected/4642c698-57ae-49ad-a5f5-54c575f0fde9-kube-api-access-7l6n7\") pod \"node-resolver-l764j\" (UID: \"4642c698-57ae-49ad-a5f5-54c575f0fde9\") " pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:16.971112 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.971089 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bhd8\" (UniqueName: \"kubernetes.io/projected/fe907bfc-2532-4c58-aacd-9b9c01bae6f7-kube-api-access-5bhd8\") pod \"multus-sq748\" (UID: \"fe907bfc-2532-4c58-aacd-9b9c01bae6f7\") " pod="openshift-multus/multus-sq748" Jul 16 12:45:16.972062 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.972039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69tv\" (UniqueName: \"kubernetes.io/projected/3c2a4180-949b-4c61-897c-9c8dbb79e831-kube-api-access-v69tv\") pod \"tuned-g4jcs\" (UID: \"3c2a4180-949b-4c61-897c-9c8dbb79e831\") " pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:16.996731 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:16.996707 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 16 12:45:17.060010 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.059978 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/743b5ace-80ef-4034-a39e-799daffaab00-host-slash\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:17.060187 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060017 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc0dd1b8-c7e2-44a2-b653-135b66d1654b-agent-certs\") pod \"konnectivity-agent-gzltq\" (UID: \"fc0dd1b8-c7e2-44a2-b653-135b66d1654b\") " pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:17.060187 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060047 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-log-socket\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060187 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060094 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/743b5ace-80ef-4034-a39e-799daffaab00-host-slash\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:17.060187 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-systemd\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060187 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060149 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-systemd\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060187 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060161 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-var-lib-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060187 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060171 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-log-socket\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060194 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-ovn\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hw24\" (UniqueName: \"kubernetes.io/projected/caf2381b-6027-4b9d-bea9-f20120aec027-kube-api-access-2hw24\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-var-lib-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060235 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-ovn\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-ovnkube-config\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060290 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcd6l\" (UniqueName: \"kubernetes.io/projected/5cf65300-74d2-4492-8245-b3984de67048-kube-api-access-kcd6l\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060341 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-run-ovn-kubernetes\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060365 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5664fccb-777b-4367-8239-458af04f6c29-host\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-ovnkube-script-lib\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-system-cni-dir\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060498 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5664fccb-777b-4367-8239-458af04f6c29-host\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060501 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkmq\" (UniqueName: \"kubernetes.io/projected/743b5ace-80ef-4034-a39e-799daffaab00-kube-api-access-gdkmq\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:17.060565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-kubelet\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-etc-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-run-ovn-kubernetes\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060657 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-etc-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zph46\" (UniqueName: \"kubernetes.io/projected/5664fccb-777b-4367-8239-458af04f6c29-kube-api-access-zph46\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060715 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-kubelet\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc0dd1b8-c7e2-44a2-b653-135b66d1654b-konnectivity-ca\") pod \"konnectivity-agent-gzltq\" (UID: \"fc0dd1b8-c7e2-44a2-b653-135b66d1654b\") " pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060760 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-run-openvswitch\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060762 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-cni-bin\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060803 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-cni-bin\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060838 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-cnibin\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsc2z\" (UniqueName: \"kubernetes.io/projected/d2c635b3-06ef-492c-967e-f051ce3a388d-kube-api-access-bsc2z\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.060959 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-systemd-units\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061040 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-slash\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.061393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-node-log\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061087 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-ovnkube-config\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/caf2381b-6027-4b9d-bea9-f20120aec027-ovn-node-metrics-cert\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5664fccb-777b-4367-8239-458af04f6c29-serviceca\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061172 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/743b5ace-80ef-4034-a39e-799daffaab00-iptables-alerter-script\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-run-netns\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.061287 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061321 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-env-overrides\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061330 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-ovnkube-script-lib\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.061358 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs podName:5cf65300-74d2-4492-8245-b3984de67048 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:17.561339838 +0000 UTC m=+3.103946563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs") pod "network-metrics-daemon-hpwb2" (UID: "5cf65300-74d2-4492-8245-b3984de67048") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061381 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-systemd-units\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061387 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-cni-netd\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061423 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-os-release\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061425 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-slash\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061654 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fc0dd1b8-c7e2-44a2-b653-135b66d1654b-konnectivity-ca\") pod \"konnectivity-agent-gzltq\" (UID: \"fc0dd1b8-c7e2-44a2-b653-135b66d1654b\") " pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061703 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-cni-netd\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-os-release\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061869 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-system-cni-dir\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-host-run-netns\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061957 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061972 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5664fccb-777b-4367-8239-458af04f6c29-serviceca\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.061963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/caf2381b-6027-4b9d-bea9-f20120aec027-node-log\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.062037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d2c635b3-06ef-492c-967e-f051ce3a388d-cnibin\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.062037 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/743b5ace-80ef-4034-a39e-799daffaab00-iptables-alerter-script\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.062188 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d2c635b3-06ef-492c-967e-f051ce3a388d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.062923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.062218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/caf2381b-6027-4b9d-bea9-f20120aec027-env-overrides\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.063266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.062928 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fc0dd1b8-c7e2-44a2-b653-135b66d1654b-agent-certs\") pod \"konnectivity-agent-gzltq\" (UID: \"fc0dd1b8-c7e2-44a2-b653-135b66d1654b\") " pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:17.063960 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.063944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/caf2381b-6027-4b9d-bea9-f20120aec027-ovn-node-metrics-cert\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.072135 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.071984 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hw24\" (UniqueName: \"kubernetes.io/projected/caf2381b-6027-4b9d-bea9-f20120aec027-kube-api-access-2hw24\") pod \"ovnkube-node-d62t6\" (UID: \"caf2381b-6027-4b9d-bea9-f20120aec027\") " pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.072862 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.072837 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkmq\" (UniqueName: \"kubernetes.io/projected/743b5ace-80ef-4034-a39e-799daffaab00-kube-api-access-gdkmq\") pod \"iptables-alerter-wb2vs\" (UID: \"743b5ace-80ef-4034-a39e-799daffaab00\") " pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:17.083491 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.083438 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsc2z\" (UniqueName: \"kubernetes.io/projected/d2c635b3-06ef-492c-967e-f051ce3a388d-kube-api-access-bsc2z\") pod \"multus-additional-cni-plugins-2kn5j\" (UID: \"d2c635b3-06ef-492c-967e-f051ce3a388d\") " pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.083491 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.083465 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zph46\" (UniqueName: \"kubernetes.io/projected/5664fccb-777b-4367-8239-458af04f6c29-kube-api-access-zph46\") pod \"node-ca-ztrvw\" (UID: \"5664fccb-777b-4367-8239-458af04f6c29\") " pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:17.083491 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.083468 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcd6l\" (UniqueName: \"kubernetes.io/projected/5cf65300-74d2-4492-8245-b3984de67048-kube-api-access-kcd6l\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:17.153996 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.153962 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-sq748" Jul 16 12:45:17.161497 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.161477 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" Jul 16 12:45:17.172135 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.172112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" Jul 16 12:45:17.177828 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.177795 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l764j" Jul 16 12:45:17.186475 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.186453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wb2vs" Jul 16 12:45:17.193645 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.193611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:17.200915 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.200892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:17.208738 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.208714 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztrvw" Jul 16 12:45:17.214522 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.214499 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" Jul 16 12:45:17.464125 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.464089 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:17.464296 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.464275 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:17.464362 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.464352 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret podName:0b6659fe-fabb-4a30-aaba-3615b0e73f11 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:18.464332826 +0000 UTC m=+4.006939554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret") pod "global-pull-secret-syncer-pvcrz" (UID: "0b6659fe-fabb-4a30-aaba-3615b0e73f11") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:17.565046 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.565013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:17.565209 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.565063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:17.565209 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.565183 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:17.565337 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.565268 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs podName:5cf65300-74d2-4492-8245-b3984de67048 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:18.565245667 +0000 UTC m=+4.107852416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs") pod "network-metrics-daemon-hpwb2" (UID: "5cf65300-74d2-4492-8245-b3984de67048") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:17.565438 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.565184 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:17.565541 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.565453 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:17.565541 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.565470 2576 projected.go:196] Error preparing data for projected volume kube-api-access-scjvm for pod openshift-network-diagnostics/network-check-target-kmxqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:17.565541 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.565520 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm podName:ad46a259-f207-45c2-8379-5040cab7d186 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:18.565508204 +0000 UTC m=+4.108114928 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-scjvm" (UniqueName: "kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm") pod "network-check-target-kmxqh" (UID: "ad46a259-f207-45c2-8379-5040cab7d186") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:17.889364 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:17.889323 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0dd1b8_c7e2_44a2_b653_135b66d1654b.slice/crio-0c6600ba16ada135ce725f82c81626e2fc75c377eff36fb9287a969b107bf157 WatchSource:0}: Error finding container 0c6600ba16ada135ce725f82c81626e2fc75c377eff36fb9287a969b107bf157: Status 404 returned error can't find the container with id 0c6600ba16ada135ce725f82c81626e2fc75c377eff36fb9287a969b107bf157 Jul 16 12:45:17.896210 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.896187 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:40:16 +0000 UTC" deadline="2028-03-13 17:49:41.564827479 +0000 UTC" Jul 16 12:45:17.896210 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.896208 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14549h4m23.668622227s" Jul 16 12:45:17.904014 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:17.903988 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf2381b_6027_4b9d_bea9_f20120aec027.slice/crio-10739ebc9c2bfb861ebbd0f1e66d89f28cd3803d95b8a02d3f4d092b68886399 WatchSource:0}: Error finding container 10739ebc9c2bfb861ebbd0f1e66d89f28cd3803d95b8a02d3f4d092b68886399: Status 404 returned error can't find the container with id 10739ebc9c2bfb861ebbd0f1e66d89f28cd3803d95b8a02d3f4d092b68886399 Jul 16 12:45:17.906317 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:17.906297 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5664fccb_777b_4367_8239_458af04f6c29.slice/crio-bae1a1600e02f83bd536baf54f46e4f255740f4ac769ebcc6a82c78507b17a47 WatchSource:0}: Error finding container bae1a1600e02f83bd536baf54f46e4f255740f4ac769ebcc6a82c78507b17a47: Status 404 returned error can't find the container with id bae1a1600e02f83bd536baf54f46e4f255740f4ac769ebcc6a82c78507b17a47 Jul 16 12:45:17.911474 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:17.911451 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743b5ace_80ef_4034_a39e_799daffaab00.slice/crio-41a5f4bcd2bcc09533870371fd4b03aa6cb2fc954749062bb1680c31604ff92b WatchSource:0}: Error finding container 41a5f4bcd2bcc09533870371fd4b03aa6cb2fc954749062bb1680c31604ff92b: Status 404 returned error can't find the container with id 41a5f4bcd2bcc09533870371fd4b03aa6cb2fc954749062bb1680c31604ff92b Jul 16 12:45:17.912537 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:17.912521 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe907bfc_2532_4c58_aacd_9b9c01bae6f7.slice/crio-058aaa21cb2745abc922d7ba3fb93e9d6e3ef493b5d6bbbce8653c0532508bc7 WatchSource:0}: Error finding container 058aaa21cb2745abc922d7ba3fb93e9d6e3ef493b5d6bbbce8653c0532508bc7: Status 404 returned error can't find the container with id 058aaa21cb2745abc922d7ba3fb93e9d6e3ef493b5d6bbbce8653c0532508bc7 Jul 16 12:45:17.913259 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:17.913211 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c635b3_06ef_492c_967e_f051ce3a388d.slice/crio-c4ff650944bdc5e9c9de9de5c7d93f45c011c9a6170e5056f05d22707b40484e WatchSource:0}: Error finding container c4ff650944bdc5e9c9de9de5c7d93f45c011c9a6170e5056f05d22707b40484e: Status 404 returned error can't find the container with id c4ff650944bdc5e9c9de9de5c7d93f45c011c9a6170e5056f05d22707b40484e Jul 16 12:45:17.980426 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.980276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:17.980531 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.980326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:17.980599 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.980527 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:17.980682 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:17.980650 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:17.987632 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.987603 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" event={"ID":"fd0fb912-647a-444a-872f-3d18d1990d56","Type":"ContainerStarted","Data":"664fcaa5791677962719529e705ff7184e82c2d5943bf940f61fb50c62f52d8a"} Jul 16 12:45:17.990928 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.990902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" event={"ID":"3c2a4180-949b-4c61-897c-9c8dbb79e831","Type":"ContainerStarted","Data":"03cd29c145be22225cf8dfa66e2107219909a5bd5f362bfd16e95a7a46a0f64f"} Jul 16 12:45:17.992227 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.992205 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztrvw" event={"ID":"5664fccb-777b-4367-8239-458af04f6c29","Type":"ContainerStarted","Data":"bae1a1600e02f83bd536baf54f46e4f255740f4ac769ebcc6a82c78507b17a47"} Jul 16 12:45:17.993182 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.993165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"10739ebc9c2bfb861ebbd0f1e66d89f28cd3803d95b8a02d3f4d092b68886399"} Jul 16 12:45:17.997115 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:17.997094 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gzltq" event={"ID":"fc0dd1b8-c7e2-44a2-b653-135b66d1654b","Type":"ContainerStarted","Data":"0c6600ba16ada135ce725f82c81626e2fc75c377eff36fb9287a969b107bf157"} Jul 16 12:45:18.004764 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.004736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" event={"ID":"0bbc06f37afebb87bdba8d241d51d6dd","Type":"ContainerStarted","Data":"cb651e33fb3b4b44926960977bd17cd46bb19066c8080b4950b97c4743b8d4b7"} Jul 16 12:45:18.007811 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.007782 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wb2vs" event={"ID":"743b5ace-80ef-4034-a39e-799daffaab00","Type":"ContainerStarted","Data":"41a5f4bcd2bcc09533870371fd4b03aa6cb2fc954749062bb1680c31604ff92b"} Jul 16 12:45:18.008850 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.008807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" event={"ID":"d2c635b3-06ef-492c-967e-f051ce3a388d","Type":"ContainerStarted","Data":"c4ff650944bdc5e9c9de9de5c7d93f45c011c9a6170e5056f05d22707b40484e"} Jul 16 12:45:18.009906 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.009884 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sq748" event={"ID":"fe907bfc-2532-4c58-aacd-9b9c01bae6f7","Type":"ContainerStarted","Data":"058aaa21cb2745abc922d7ba3fb93e9d6e3ef493b5d6bbbce8653c0532508bc7"} Jul 16 12:45:18.011273 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.011247 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l764j" event={"ID":"4642c698-57ae-49ad-a5f5-54c575f0fde9","Type":"ContainerStarted","Data":"bf6b2eadb7bdeff24f37e04cd599092afaebe78cf3b5022f903e2710b8848b34"} Jul 16 12:45:18.023410 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.023361 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-160.ec2.internal" podStartSLOduration=2.023345669 podStartE2EDuration="2.023345669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:18.023105216 +0000 UTC m=+3.565711964" watchObservedRunningTime="2026-07-16 12:45:18.023345669 +0000 UTC m=+3.565952417" Jul 16 12:45:18.472415 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.472333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:18.472573 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.472487 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:18.472573 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.472550 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret podName:0b6659fe-fabb-4a30-aaba-3615b0e73f11 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:20.472532649 +0000 UTC m=+6.015139375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret") pod "global-pull-secret-syncer-pvcrz" (UID: "0b6659fe-fabb-4a30-aaba-3615b0e73f11") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:18.573347 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.573310 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:18.573524 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.573378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:18.573524 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.573513 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:18.573634 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.573537 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:18.573634 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.573549 2576 projected.go:196] Error preparing data for projected volume kube-api-access-scjvm for pod openshift-network-diagnostics/network-check-target-kmxqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:18.573634 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.573607 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm podName:ad46a259-f207-45c2-8379-5040cab7d186 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:20.573586991 +0000 UTC m=+6.116193734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-scjvm" (UniqueName: "kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm") pod "network-check-target-kmxqh" (UID: "ad46a259-f207-45c2-8379-5040cab7d186") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:18.574032 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.574013 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:18.574143 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.574091 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs podName:5cf65300-74d2-4492-8245-b3984de67048 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:20.574073856 +0000 UTC m=+6.116680597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs") pod "network-metrics-daemon-hpwb2" (UID: "5cf65300-74d2-4492-8245-b3984de67048") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:18.896969 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.896855 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 12:40:16 +0000 UTC" deadline="2028-01-25 12:57:42.952027382 +0000 UTC" Jul 16 12:45:18.896969 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.896915 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13392h12m24.055115923s" Jul 16 12:45:18.981721 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:18.981691 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:18.981927 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:18.981859 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:19.026045 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:19.026008 2576 generic.go:356] "Generic (PLEG): container finished" podID="02c19a6ca1b613e10d3b866f18f227c8" containerID="f21ec385997ce7bfa693b0968a6b0f13162591cb6f8919ee0255e231620b8e65" exitCode=0 Jul 16 12:45:19.026308 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:19.026218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" event={"ID":"02c19a6ca1b613e10d3b866f18f227c8","Type":"ContainerDied","Data":"f21ec385997ce7bfa693b0968a6b0f13162591cb6f8919ee0255e231620b8e65"} Jul 16 12:45:19.980882 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:19.980833 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:19.981517 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:19.981009 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:19.981517 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:19.981508 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:19.981650 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:19.981624 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:20.046097 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:20.046057 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" event={"ID":"02c19a6ca1b613e10d3b866f18f227c8","Type":"ContainerStarted","Data":"01001410ab21c26a40a44e3411a576dce01fcec43435853c3040bb30fd154b9a"} Jul 16 12:45:20.489245 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:20.489210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:20.489425 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.489378 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:20.489425 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.489442 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret podName:0b6659fe-fabb-4a30-aaba-3615b0e73f11 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:24.489423869 +0000 UTC m=+10.032030597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret") pod "global-pull-secret-syncer-pvcrz" (UID: "0b6659fe-fabb-4a30-aaba-3615b0e73f11") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:20.590625 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:20.590587 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:20.590799 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:20.590666 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:20.590883 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.590848 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:20.590883 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.590868 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:20.590883 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.590883 2576 projected.go:196] Error preparing data for projected volume kube-api-access-scjvm for pod openshift-network-diagnostics/network-check-target-kmxqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:20.591047 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.590941 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm podName:ad46a259-f207-45c2-8379-5040cab7d186 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:24.590922994 +0000 UTC m=+10.133529725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-scjvm" (UniqueName: "kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm") pod "network-check-target-kmxqh" (UID: "ad46a259-f207-45c2-8379-5040cab7d186") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:20.591047 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.591016 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:20.591154 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.591049 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs podName:5cf65300-74d2-4492-8245-b3984de67048 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:24.591038113 +0000 UTC m=+10.133644840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs") pod "network-metrics-daemon-hpwb2" (UID: "5cf65300-74d2-4492-8245-b3984de67048") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:20.980985 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:20.980910 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:20.981424 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:20.981043 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:21.980751 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:21.980717 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:21.980951 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:21.980868 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:21.981228 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:21.981205 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:21.981588 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:21.981302 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:22.982053 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:22.982021 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:22.982525 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:22.982164 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:23.980895 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:23.980833 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:23.980895 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:23.980863 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:23.981136 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:23.980980 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:23.981136 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:23.981110 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:24.524765 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:24.524686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:24.525249 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.524864 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:24.525249 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.524927 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret podName:0b6659fe-fabb-4a30-aaba-3615b0e73f11 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:32.524907744 +0000 UTC m=+18.067514486 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret") pod "global-pull-secret-syncer-pvcrz" (UID: "0b6659fe-fabb-4a30-aaba-3615b0e73f11") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:24.625206 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:24.625106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:24.625206 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:24.625196 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:24.625455 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.625272 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:24.625455 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.625298 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:24.625455 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.625312 2576 projected.go:196] Error preparing data for projected volume kube-api-access-scjvm for pod openshift-network-diagnostics/network-check-target-kmxqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:24.625455 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.625337 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:24.625455 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.625369 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm podName:ad46a259-f207-45c2-8379-5040cab7d186 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:32.625351309 +0000 UTC m=+18.167958040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-scjvm" (UniqueName: "kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm") pod "network-check-target-kmxqh" (UID: "ad46a259-f207-45c2-8379-5040cab7d186") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:24.625455 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.625391 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs podName:5cf65300-74d2-4492-8245-b3984de67048 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:32.625379308 +0000 UTC m=+18.167986034 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs") pod "network-metrics-daemon-hpwb2" (UID: "5cf65300-74d2-4492-8245-b3984de67048") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:24.982193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:24.982109 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:24.982352 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:24.982227 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:25.981281 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:25.980667 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:25.981281 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:25.980800 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:25.981281 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:25.981145 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:25.981281 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:25.981245 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:26.980473 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:26.980435 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:26.980662 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:26.980561 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:27.980256 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:27.980209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:27.980256 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:27.980253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:27.980793 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:27.980332 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:27.980793 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:27.980463 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:28.981005 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:28.980972 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:28.981456 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:28.981107 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:29.980844 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:29.980785 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:29.981031 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:29.980970 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:29.981031 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:29.981004 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:29.981395 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:29.981124 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:30.980747 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:30.980718 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:30.980918 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:30.980844 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:31.980963 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:31.980931 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:31.981429 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:31.980979 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:31.981429 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:31.981059 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:31.981429 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:31.981202 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:32.588234 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:32.588201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:32.588420 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.588321 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:32.588420 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.588384 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret podName:0b6659fe-fabb-4a30-aaba-3615b0e73f11 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:48.588365535 +0000 UTC m=+34.130972259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret") pod "global-pull-secret-syncer-pvcrz" (UID: "0b6659fe-fabb-4a30-aaba-3615b0e73f11") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:32.688440 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:32.688411 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:32.688579 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:32.688463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:32.688579 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.688552 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:32.688650 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.688593 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:32.688650 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.688610 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:32.688650 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.688619 2576 projected.go:196] Error preparing data for projected volume kube-api-access-scjvm for pod openshift-network-diagnostics/network-check-target-kmxqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:32.688650 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.688609 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs podName:5cf65300-74d2-4492-8245-b3984de67048 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:48.688594426 +0000 UTC m=+34.231201151 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs") pod "network-metrics-daemon-hpwb2" (UID: "5cf65300-74d2-4492-8245-b3984de67048") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:32.688807 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.688667 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm podName:ad46a259-f207-45c2-8379-5040cab7d186 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:48.688652124 +0000 UTC m=+34.231258855 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-scjvm" (UniqueName: "kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm") pod "network-check-target-kmxqh" (UID: "ad46a259-f207-45c2-8379-5040cab7d186") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:32.980959 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:32.980930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:32.981142 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:32.981043 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:33.980966 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:33.980877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:33.980966 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:33.980896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:33.981489 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:33.981023 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:33.981489 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:33.981111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:34.981976 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:34.981937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:34.982425 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:34.982385 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:35.980609 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:35.980577 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:35.980794 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:35.980710 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:35.980794 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:35.980767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:35.980932 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:35.980879 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:36.981258 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:36.981042 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:36.981797 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:36.981370 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:37.078726 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.078698 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gzltq" event={"ID":"fc0dd1b8-c7e2-44a2-b653-135b66d1654b","Type":"ContainerStarted","Data":"624b9a860dfc5152ea118e1dad4c93e0e726e7a353d99e70b348503b004d9c13"} Jul 16 12:45:37.080506 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.080476 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2c635b3-06ef-492c-967e-f051ce3a388d" containerID="33f2e52a81e88e3874d7d294c89b35cba65b50b00b4fc2f1c84257d4f4da8b67" exitCode=0 Jul 16 12:45:37.080615 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.080556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" event={"ID":"d2c635b3-06ef-492c-967e-f051ce3a388d","Type":"ContainerDied","Data":"33f2e52a81e88e3874d7d294c89b35cba65b50b00b4fc2f1c84257d4f4da8b67"} Jul 16 12:45:37.082943 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.082917 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-sq748" event={"ID":"fe907bfc-2532-4c58-aacd-9b9c01bae6f7","Type":"ContainerStarted","Data":"310a7a67d538a4459e5b09e4787a9545bafd6b180edcea1a280a8d828b58c293"} Jul 16 12:45:37.084476 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.084450 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l764j" event={"ID":"4642c698-57ae-49ad-a5f5-54c575f0fde9","Type":"ContainerStarted","Data":"3ca4389fd2d544d070116badb2c708638de924c5dcdf602152de9d1ef1e2ac53"} Jul 16 12:45:37.085599 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.085581 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" event={"ID":"fd0fb912-647a-444a-872f-3d18d1990d56","Type":"ContainerStarted","Data":"1077db43d9ca37235dad85d873c666330da130e8651c731e554a1f2410be2251"} Jul 16 12:45:37.086900 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.086881 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" event={"ID":"3c2a4180-949b-4c61-897c-9c8dbb79e831","Type":"ContainerStarted","Data":"8723ac70d0e4fc268b8bbe1a7d4af5dfac3cfb6c7d3bd507be06ecde5f65dd96"} Jul 16 12:45:37.088198 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.088178 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztrvw" event={"ID":"5664fccb-777b-4367-8239-458af04f6c29","Type":"ContainerStarted","Data":"19be170cf269bf19d01483e6f68ea2c06fc84f26b0a41a30e8c733cf91ba353d"} Jul 16 12:45:37.089741 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.089724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"e523e35fec6600a57a45520b96ef73344c8479b0cac8762206fb2329af9e956e"} Jul 16 12:45:37.089799 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.089772 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"c8bb72f0b97fcd8ae522fb90b19c01a00d384503074e16b2f614f04bdaef9077"} Jul 16 12:45:37.098714 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.098677 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-160.ec2.internal" podStartSLOduration=21.0986514 podStartE2EDuration="21.0986514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:20.066865749 +0000 UTC m=+5.609472494" watchObservedRunningTime="2026-07-16 12:45:37.0986514 +0000 UTC m=+22.641258140" Jul 16 12:45:37.099218 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.099190 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gzltq" podStartSLOduration=3.492035853 podStartE2EDuration="22.099180284s" totalImagesPullingTime="18.607144431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:37.098266274 +0000 UTC m=+22.640873020" watchObservedRunningTime="2026-07-16 12:45:37.099180284 +0000 UTC m=+22.641787031" Jul 16 12:45:37.155942 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.155905 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-sq748" podStartSLOduration=3.279400094 podStartE2EDuration="22.155892621s" totalImagesPullingTime="18.876492527s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:37.155539349 +0000 UTC m=+22.698146097" watchObservedRunningTime="2026-07-16 12:45:37.155892621 +0000 UTC m=+22.698499368" Jul 16 12:45:37.179506 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.179470 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-l764j" podStartSLOduration=3.544218308 podStartE2EDuration="22.179457185s" totalImagesPullingTime="18.635238877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:37.179397044 +0000 UTC m=+22.722003791" watchObservedRunningTime="2026-07-16 12:45:37.179457185 +0000 UTC m=+22.722063932" Jul 16 12:45:37.195719 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.195672 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ztrvw" podStartSLOduration=3.559443039 podStartE2EDuration="22.195659135s" totalImagesPullingTime="18.636216096s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:37.195655542 +0000 UTC m=+22.738262302" watchObservedRunningTime="2026-07-16 12:45:37.195659135 +0000 UTC m=+22.738265876" Jul 16 12:45:37.219609 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.219565 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g4jcs" podStartSLOduration=3.583699549 podStartE2EDuration="22.219546044s" totalImagesPullingTime="18.635846495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:37.21917138 +0000 UTC m=+22.761778119" watchObservedRunningTime="2026-07-16 12:45:37.219546044 +0000 UTC m=+22.762152791" Jul 16 12:45:37.980793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.980597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:37.980991 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:37.980598 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:37.980991 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:37.980915 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:37.980991 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:37.980961 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:38.094790 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.094767 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 12:45:38.095372 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.095231 2576 generic.go:356] "Generic (PLEG): container finished" podID="caf2381b-6027-4b9d-bea9-f20120aec027" containerID="e523e35fec6600a57a45520b96ef73344c8479b0cac8762206fb2329af9e956e" exitCode=1 Jul 16 12:45:38.095372 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.095301 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerDied","Data":"e523e35fec6600a57a45520b96ef73344c8479b0cac8762206fb2329af9e956e"} Jul 16 12:45:38.095485 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.095386 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"88316c2145d4510b769203e5fdcb8f72e896287ee944ad87b208873175bc1ecc"} Jul 16 12:45:38.095485 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.095438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"895ed10aa725001304357747e58e430a5331a3449e2b160cb4794c92abb5c8e2"} Jul 16 12:45:38.095589 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.095486 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"4fbd85b69bff6ebc013043799beee4b6d67d8d93764cbc2367613561a222c765"} Jul 16 12:45:38.095589 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.095533 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"2bde8545f3dae705cc0805c40327e13a3ceb6bde464cf9a6a95d956d029746ca"} Jul 16 12:45:38.097118 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.097084 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wb2vs" event={"ID":"743b5ace-80ef-4034-a39e-799daffaab00","Type":"ContainerStarted","Data":"6024b290c6e28ff981fdcdd0949039da64dbb91810c0b7deb1e695dc4403c8b5"} Jul 16 12:45:38.117664 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.117579 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wb2vs" podStartSLOduration=4.479399855 podStartE2EDuration="23.11756263s" totalImagesPullingTime="18.638162775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:38.11699715 +0000 UTC m=+23.659603899" watchObservedRunningTime="2026-07-16 12:45:38.11756263 +0000 UTC m=+23.660169378" Jul 16 12:45:38.980929 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:38.980895 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:38.981208 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:38.981028 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:39.807914 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:39.807740 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 16 12:45:39.910423 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:39.910320 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T12:45:39.807908049Z","UUID":"c32ddd16-140a-4c29-ba9e-fef8d7823e6f","Handler":null,"Name":"","Endpoint":""} Jul 16 12:45:39.911686 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:39.911666 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 16 12:45:39.911790 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:39.911698 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 16 12:45:39.980727 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:39.980703 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:39.980903 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:39.980708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:39.980966 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:39.980790 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:39.980966 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:39.980921 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:40.105947 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:40.105874 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 12:45:40.106328 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:40.106288 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"785e0d846c2a26ba6408424665ea8e8afc0797d2a626937d609d8df776fcded4"} Jul 16 12:45:40.108298 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:40.108276 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" event={"ID":"fd0fb912-647a-444a-872f-3d18d1990d56","Type":"ContainerStarted","Data":"abebd5140ecbe8ba54fc2892ca09f623f2b4d49edb45d987ca7bbd8dcdd333bb"} Jul 16 12:45:40.980894 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:40.980863 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:40.981423 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:40.980983 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:41.930063 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:41.929803 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:41.931029 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:41.930739 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:41.980202 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:41.980070 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:41.980302 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:41.980075 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:41.980338 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:41.980298 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:41.980392 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:41.980341 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:42.117505 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.117401 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 12:45:42.118350 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.117810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"cdae25ce21c0c4f6924fce71e417a7c5e09fade542eda35ee747e67e97b1cfd5"} Jul 16 12:45:42.118350 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.118088 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:42.118350 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.118325 2576 scope.go:117] "RemoveContainer" containerID="e523e35fec6600a57a45520b96ef73344c8479b0cac8762206fb2329af9e956e" Jul 16 12:45:42.118575 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.118484 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:42.118666 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.118638 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:42.120325 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.120299 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2c635b3-06ef-492c-967e-f051ce3a388d" containerID="ea3c96ae45b4f32311907d2dd523e140a7c41fba983bb23148370169a81d641c" exitCode=0 Jul 16 12:45:42.120450 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.120404 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" event={"ID":"d2c635b3-06ef-492c-967e-f051ce3a388d","Type":"ContainerDied","Data":"ea3c96ae45b4f32311907d2dd523e140a7c41fba983bb23148370169a81d641c"} Jul 16 12:45:42.138566 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.138544 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:42.138683 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.138673 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:45:42.981069 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:42.981000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:42.981221 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:42.981101 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:43.124676 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.124636 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" event={"ID":"fd0fb912-647a-444a-872f-3d18d1990d56","Type":"ContainerStarted","Data":"738b5b9c985bbe05314a29e44fc0e2d4afbd40d507a7fa33876bd16f3e0c9711"} Jul 16 12:45:43.132690 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.132665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 12:45:43.133124 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.133097 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" event={"ID":"caf2381b-6027-4b9d-bea9-f20120aec027","Type":"ContainerStarted","Data":"b46ba82332584ecaa3ff89885353836a81f93646b008b023feca8576ea09badb"} Jul 16 12:45:43.150768 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.150722 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tzggf" podStartSLOduration=3.792268694 podStartE2EDuration="28.150709079s" totalImagesPullingTime="24.358440385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:43.149159311 +0000 UTC m=+28.691766070" watchObservedRunningTime="2026-07-16 12:45:43.150709079 +0000 UTC m=+28.693315824" Jul 16 12:45:43.872931 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.872866 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" podStartSLOduration=9.957420471 podStartE2EDuration="28.872848652s" totalImagesPullingTime="18.915428181s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:43.180276634 +0000 UTC m=+28.722883395" watchObservedRunningTime="2026-07-16 12:45:43.872848652 +0000 UTC m=+29.415455398" Jul 16 12:45:43.873125 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.873107 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pvcrz"] Jul 16 12:45:43.873225 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.873213 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:43.873361 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:43.873326 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:43.873872 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.873847 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kmxqh"] Jul 16 12:45:43.873967 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.873932 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:43.874011 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:43.873999 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:43.886095 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.886069 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hpwb2"] Jul 16 12:45:43.886212 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:43.886152 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:43.886275 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:43.886227 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:44.019375 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:44.019299 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:44.019508 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:44.019433 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 16 12:45:44.019803 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:44.019774 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gzltq" Jul 16 12:45:44.981013 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:44.980786 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:44.981432 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:44.980864 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:44.981432 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:44.981025 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:44.981432 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:44.981114 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:45.139148 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:45.139113 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2c635b3-06ef-492c-967e-f051ce3a388d" containerID="77903639a322648b5413a8ee22b7b0b808b7b070c5381897d0b522ea9e60784e" exitCode=0 Jul 16 12:45:45.139296 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:45.139184 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" event={"ID":"d2c635b3-06ef-492c-967e-f051ce3a388d","Type":"ContainerDied","Data":"77903639a322648b5413a8ee22b7b0b808b7b070c5381897d0b522ea9e60784e"} Jul 16 12:45:45.980866 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:45.980833 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:45.981008 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:45.980946 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:46.980995 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:46.980970 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:46.981550 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:46.981085 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:46.981550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:46.981158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:46.981550 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:46.981257 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:47.145833 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:47.145790 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2c635b3-06ef-492c-967e-f051ce3a388d" containerID="9d2f46d3c15423b95b077784c08e1415bd9320a21480c78d6536abf4532d8ee4" exitCode=0 Jul 16 12:45:47.145982 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:47.145854 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" event={"ID":"d2c635b3-06ef-492c-967e-f051ce3a388d","Type":"ContainerDied","Data":"9d2f46d3c15423b95b077784c08e1415bd9320a21480c78d6536abf4532d8ee4"} Jul 16 12:45:47.980908 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:47.980877 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:47.981047 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:47.980980 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:48.607189 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:48.607153 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:48.607358 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.607330 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:48.607423 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.607399 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret podName:0b6659fe-fabb-4a30-aaba-3615b0e73f11 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:20.607377408 +0000 UTC m=+66.149984152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret") pod "global-pull-secret-syncer-pvcrz" (UID: "0b6659fe-fabb-4a30-aaba-3615b0e73f11") : object "kube-system"/"original-pull-secret" not registered Jul 16 12:45:48.707733 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:48.707705 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:48.707923 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:48.707749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:48.707923 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.707869 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 16 12:45:48.707923 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.707884 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 16 12:45:48.707923 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.707893 2576 projected.go:196] Error preparing data for projected volume kube-api-access-scjvm for pod openshift-network-diagnostics/network-check-target-kmxqh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:48.708135 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.707933 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm podName:ad46a259-f207-45c2-8379-5040cab7d186 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:20.707920537 +0000 UTC m=+66.250527262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-scjvm" (UniqueName: "kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm") pod "network-check-target-kmxqh" (UID: "ad46a259-f207-45c2-8379-5040cab7d186") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 16 12:45:48.708135 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.707869 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:48.708135 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.708004 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs podName:5cf65300-74d2-4492-8245-b3984de67048 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:20.707986166 +0000 UTC m=+66.250592911 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs") pod "network-metrics-daemon-hpwb2" (UID: "5cf65300-74d2-4492-8245-b3984de67048") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 16 12:45:48.980828 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:48.980634 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:48.981005 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:48.980650 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:48.981005 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.980909 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-pvcrz" podUID="0b6659fe-fabb-4a30-aaba-3615b0e73f11" Jul 16 12:45:48.981359 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:48.981021 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hpwb2" podUID="5cf65300-74d2-4492-8245-b3984de67048" Jul 16 12:45:49.980947 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:49.980916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:49.981147 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:49.981024 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kmxqh" podUID="ad46a259-f207-45c2-8379-5040cab7d186" Jul 16 12:45:50.324288 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.324220 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-160.ec2.internal" event="NodeReady" Jul 16 12:45:50.324426 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.324346 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 16 12:45:50.390460 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.390428 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8l5pc"] Jul 16 12:45:50.392830 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.392802 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.395405 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.395385 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-f4c78"] Jul 16 12:45:50.395946 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.395835 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 16 12:45:50.395946 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.395872 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lnsbk\"" Jul 16 12:45:50.396086 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.395872 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 16 12:45:50.396971 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.396954 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f4c78" Jul 16 12:45:50.399735 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.399720 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 16 12:45:50.399952 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.399884 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wlnmp\"" Jul 16 12:45:50.400088 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.400073 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.400436 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.400416 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.409860 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.409841 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8l5pc"] Jul 16 12:45:50.412021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.411997 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f4c78"] Jul 16 12:45:50.467684 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.467654 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492"] Jul 16 12:45:50.469471 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.469454 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865"] Jul 16 12:45:50.469597 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.469582 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492" Jul 16 12:45:50.473963 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.473274 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.473963 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.473564 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2nqqj\"" Jul 16 12:45:50.473963 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.473779 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.474213 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.474153 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz"] Jul 16 12:45:50.474267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.474206 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865" Jul 16 12:45:50.476421 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.476401 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58fc699ffc-smbfr"] Jul 16 12:45:50.476558 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.476533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.477106 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.477089 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-t68sq\"" Jul 16 12:45:50.477660 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.477640 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.477762 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.477704 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.478292 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.478018 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.481138 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.481121 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 16 12:45:50.482278 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.482258 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 16 12:45:50.482809 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.482535 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.482809 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.482579 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rr49k\"" Jul 16 12:45:50.482809 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.482645 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.483011 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.482982 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 16 12:45:50.483082 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.483063 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 16 12:45:50.483132 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.483069 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-ljl5q\"" Jul 16 12:45:50.484115 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.484094 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 16 12:45:50.488247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.488218 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 16 12:45:50.493406 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.493384 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865"] Jul 16 12:45:50.494364 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.494345 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492"] Jul 16 12:45:50.497493 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.497446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz"] Jul 16 12:45:50.498354 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.498336 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58fc699ffc-smbfr"] Jul 16 12:45:50.520645 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.520625 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qw5\" (UniqueName: \"kubernetes.io/projected/06467239-8bf8-4732-a273-e283e67af2dc-kube-api-access-p5qw5\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.520754 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.520679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38cf452b-ac67-4fcb-bfa3-312e08ac1687-cert\") pod \"ingress-canary-f4c78\" (UID: \"38cf452b-ac67-4fcb-bfa3-312e08ac1687\") " pod="openshift-ingress-canary/ingress-canary-f4c78" Jul 16 12:45:50.520754 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.520724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psjg6\" (UniqueName: \"kubernetes.io/projected/38cf452b-ac67-4fcb-bfa3-312e08ac1687-kube-api-access-psjg6\") pod \"ingress-canary-f4c78\" (UID: \"38cf452b-ac67-4fcb-bfa3-312e08ac1687\") " pod="openshift-ingress-canary/ingress-canary-f4c78" Jul 16 12:45:50.520849 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.520754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06467239-8bf8-4732-a273-e283e67af2dc-tmp-dir\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.520849 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.520828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06467239-8bf8-4732-a273-e283e67af2dc-config-volume\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.520937 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.520850 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.569446 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.569415 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-dswpk"] Jul 16 12:45:50.571351 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.571328 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.575214 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.575126 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 16 12:45:50.575214 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.575176 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 16 12:45:50.575214 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.575188 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-plqbd\"" Jul 16 12:45:50.575433 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.575384 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:45:50.576220 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.576194 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 16 12:45:50.581874 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.581855 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 16 12:45:50.592435 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.592408 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-dswpk"] Jul 16 12:45:50.621934 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.621901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjnfz\" (UniqueName: \"kubernetes.io/projected/75a27340-e369-4927-80d0-1f9b6cf7236d-kube-api-access-gjnfz\") pod \"network-check-source-869dcfcdcd-zr865\" (UID: \"75a27340-e369-4927-80d0-1f9b6cf7236d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865" Jul 16 12:45:50.622065 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.621946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvq66\" (UniqueName: \"kubernetes.io/projected/192c0582-645d-451e-8f1a-edef1f89bbbd-kube-api-access-nvq66\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.622065 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.621987 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38cf452b-ac67-4fcb-bfa3-312e08ac1687-cert\") pod \"ingress-canary-f4c78\" (UID: \"38cf452b-ac67-4fcb-bfa3-312e08ac1687\") " pod="openshift-ingress-canary/ingress-canary-f4c78" Jul 16 12:45:50.622065 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622039 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psjg6\" (UniqueName: \"kubernetes.io/projected/38cf452b-ac67-4fcb-bfa3-312e08ac1687-kube-api-access-psjg6\") pod \"ingress-canary-f4c78\" (UID: \"38cf452b-ac67-4fcb-bfa3-312e08ac1687\") " pod="openshift-ingress-canary/ingress-canary-f4c78" Jul 16 12:45:50.622218 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622190 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06467239-8bf8-4732-a273-e283e67af2dc-tmp-dir\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.622267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-ca-trust-extracted\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.622318 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-bound-sa-token\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.622355 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/192c0582-645d-451e-8f1a-edef1f89bbbd-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.622403 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06467239-8bf8-4732-a273-e283e67af2dc-config-volume\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.622453 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-trusted-ca\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.622453 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhgm\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-kube-api-access-vwhgm\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.622453 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622442 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4r8d\" (UniqueName: \"kubernetes.io/projected/db1a9477-cd53-41f9-b36c-b9831895cca0-kube-api-access-x4r8d\") pod \"volume-data-source-validator-78495597f6-gn492\" (UID: \"db1a9477-cd53-41f9-b36c-b9831895cca0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492" Jul 16 12:45:50.622599 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.622599 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622512 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-certificates\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.622599 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622530 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/192c0582-645d-451e-8f1a-edef1f89bbbd-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.622599 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622561 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-image-registry-private-configuration\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.622778 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:50.622614 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:50.622778 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622608 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.622778 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:50.622678 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls podName:06467239-8bf8-4732-a273-e283e67af2dc nodeName:}" failed. No retries permitted until 2026-07-16 12:45:51.122658556 +0000 UTC m=+36.665265281 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls") pod "dns-default-8l5pc" (UID: "06467239-8bf8-4732-a273-e283e67af2dc") : secret "dns-default-metrics-tls" not found Jul 16 12:45:50.622778 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-installation-pull-secrets\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.622778 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.622721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qw5\" (UniqueName: \"kubernetes.io/projected/06467239-8bf8-4732-a273-e283e67af2dc-kube-api-access-p5qw5\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.623160 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.623137 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06467239-8bf8-4732-a273-e283e67af2dc-tmp-dir\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.623385 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.623365 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06467239-8bf8-4732-a273-e283e67af2dc-config-volume\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.627658 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.627635 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/38cf452b-ac67-4fcb-bfa3-312e08ac1687-cert\") pod \"ingress-canary-f4c78\" (UID: \"38cf452b-ac67-4fcb-bfa3-312e08ac1687\") " pod="openshift-ingress-canary/ingress-canary-f4c78" Jul 16 12:45:50.634566 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.634544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qw5\" (UniqueName: \"kubernetes.io/projected/06467239-8bf8-4732-a273-e283e67af2dc-kube-api-access-p5qw5\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:50.634649 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.634623 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psjg6\" (UniqueName: \"kubernetes.io/projected/38cf452b-ac67-4fcb-bfa3-312e08ac1687-kube-api-access-psjg6\") pod \"ingress-canary-f4c78\" (UID: \"38cf452b-ac67-4fcb-bfa3-312e08ac1687\") " pod="openshift-ingress-canary/ingress-canary-f4c78" Jul 16 12:45:50.709613 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.709575 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-f4c78" Jul 16 12:45:50.723925 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.723895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-certificates\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724036 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.723940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/192c0582-645d-451e-8f1a-edef1f89bbbd-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.724036 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.723972 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-image-registry-private-configuration\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724036 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.723999 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724036 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724022 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-installation-pull-secrets\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724249 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1441a03-825b-4cb2-8d29-35bfd66d04f9-serving-cert\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.724249 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724100 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjnfz\" (UniqueName: \"kubernetes.io/projected/75a27340-e369-4927-80d0-1f9b6cf7236d-kube-api-access-gjnfz\") pod \"network-check-source-869dcfcdcd-zr865\" (UID: \"75a27340-e369-4927-80d0-1f9b6cf7236d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865" Jul 16 12:45:50.724249 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724125 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvq66\" (UniqueName: \"kubernetes.io/projected/192c0582-645d-451e-8f1a-edef1f89bbbd-kube-api-access-nvq66\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.724249 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-ca-trust-extracted\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724249 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-bound-sa-token\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724449 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g89vr\" (UniqueName: \"kubernetes.io/projected/f1441a03-825b-4cb2-8d29-35bfd66d04f9-kube-api-access-g89vr\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.724449 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/192c0582-645d-451e-8f1a-edef1f89bbbd-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.724449 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1441a03-825b-4cb2-8d29-35bfd66d04f9-trusted-ca\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.724449 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724354 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1441a03-825b-4cb2-8d29-35bfd66d04f9-config\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.724449 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724385 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-trusted-ca\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724449 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhgm\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-kube-api-access-vwhgm\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724449 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4r8d\" (UniqueName: \"kubernetes.io/projected/db1a9477-cd53-41f9-b36c-b9831895cca0-kube-api-access-x4r8d\") pod \"volume-data-source-validator-78495597f6-gn492\" (UID: \"db1a9477-cd53-41f9-b36c-b9831895cca0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492" Jul 16 12:45:50.724659 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-certificates\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.724659 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.724560 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/192c0582-645d-451e-8f1a-edef1f89bbbd-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.724659 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:50.724593 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:50.724659 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:50.724607 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fc699ffc-smbfr: secret "image-registry-tls" not found Jul 16 12:45:50.724659 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:50.724654 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls podName:2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:51.224641574 +0000 UTC m=+36.767248302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls") pod "image-registry-58fc699ffc-smbfr" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71") : secret "image-registry-tls" not found Jul 16 12:45:50.725445 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.725401 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-ca-trust-extracted\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.725544 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.725499 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-trusted-ca\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.727032 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.726947 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/192c0582-645d-451e-8f1a-edef1f89bbbd-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.727032 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.727018 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-image-registry-private-configuration\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.727573 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.727550 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-installation-pull-secrets\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.738347 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.738321 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4r8d\" (UniqueName: \"kubernetes.io/projected/db1a9477-cd53-41f9-b36c-b9831895cca0-kube-api-access-x4r8d\") pod \"volume-data-source-validator-78495597f6-gn492\" (UID: \"db1a9477-cd53-41f9-b36c-b9831895cca0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492" Jul 16 12:45:50.738501 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.738474 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-bound-sa-token\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.738501 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.738492 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjnfz\" (UniqueName: \"kubernetes.io/projected/75a27340-e369-4927-80d0-1f9b6cf7236d-kube-api-access-gjnfz\") pod \"network-check-source-869dcfcdcd-zr865\" (UID: \"75a27340-e369-4927-80d0-1f9b6cf7236d\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865" Jul 16 12:45:50.738884 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.738863 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhgm\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-kube-api-access-vwhgm\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:50.739178 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.739156 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvq66\" (UniqueName: \"kubernetes.io/projected/192c0582-645d-451e-8f1a-edef1f89bbbd-kube-api-access-nvq66\") pod \"kube-storage-version-migrator-operator-799cb7f958-6cmbz\" (UID: \"192c0582-645d-451e-8f1a-edef1f89bbbd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.784200 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.784171 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492" Jul 16 12:45:50.792004 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.791980 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865" Jul 16 12:45:50.801025 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.801006 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" Jul 16 12:45:50.825289 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.825213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1441a03-825b-4cb2-8d29-35bfd66d04f9-serving-cert\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.825426 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.825291 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g89vr\" (UniqueName: \"kubernetes.io/projected/f1441a03-825b-4cb2-8d29-35bfd66d04f9-kube-api-access-g89vr\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.825426 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.825335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1441a03-825b-4cb2-8d29-35bfd66d04f9-trusted-ca\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.825426 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.825359 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1441a03-825b-4cb2-8d29-35bfd66d04f9-config\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.826100 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.826069 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f1441a03-825b-4cb2-8d29-35bfd66d04f9-config\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.826309 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.826288 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1441a03-825b-4cb2-8d29-35bfd66d04f9-trusted-ca\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.828082 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.828061 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f1441a03-825b-4cb2-8d29-35bfd66d04f9-serving-cert\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.838676 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.838653 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g89vr\" (UniqueName: \"kubernetes.io/projected/f1441a03-825b-4cb2-8d29-35bfd66d04f9-kube-api-access-g89vr\") pod \"console-operator-7866497494-dswpk\" (UID: \"f1441a03-825b-4cb2-8d29-35bfd66d04f9\") " pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.880587 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.880545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:45:50.980674 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.980643 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:45:50.980852 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.980835 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:45:50.984301 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.984270 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:45:50.984301 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.984298 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xrlr5\"" Jul 16 12:45:50.984682 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:50.984614 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:45:51.127722 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.127640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:51.127902 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:51.127754 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:51.127902 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:51.127899 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls podName:06467239-8bf8-4732-a273-e283e67af2dc nodeName:}" failed. No retries permitted until 2026-07-16 12:45:52.127880327 +0000 UTC m=+37.670487054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls") pod "dns-default-8l5pc" (UID: "06467239-8bf8-4732-a273-e283e67af2dc") : secret "dns-default-metrics-tls" not found Jul 16 12:45:51.229470 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.228962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:51.229470 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:51.229178 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:51.229470 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:51.229195 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fc699ffc-smbfr: secret "image-registry-tls" not found Jul 16 12:45:51.229470 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:51.229251 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls podName:2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:52.229234212 +0000 UTC m=+37.771840956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls") pod "image-registry-58fc699ffc-smbfr" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71") : secret "image-registry-tls" not found Jul 16 12:45:51.455083 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.454874 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492"] Jul 16 12:45:51.460353 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:51.460311 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1a9477_cd53_41f9_b36c_b9831895cca0.slice/crio-c8105d496715a1fd59c9dd444aca12d5df4e6f0bb63920e8158bafacb1e1c1d6 WatchSource:0}: Error finding container c8105d496715a1fd59c9dd444aca12d5df4e6f0bb63920e8158bafacb1e1c1d6: Status 404 returned error can't find the container with id c8105d496715a1fd59c9dd444aca12d5df4e6f0bb63920e8158bafacb1e1c1d6 Jul 16 12:45:51.467751 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.467709 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-dswpk"] Jul 16 12:45:51.470321 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.470251 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz"] Jul 16 12:45:51.471740 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:51.471716 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1441a03_825b_4cb2_8d29_35bfd66d04f9.slice/crio-aa1acca30d71dd032ef8d368a4fe1456e0f18f4bd4e587d7ec00f7a41c0de9d6 WatchSource:0}: Error finding container aa1acca30d71dd032ef8d368a4fe1456e0f18f4bd4e587d7ec00f7a41c0de9d6: Status 404 returned error can't find the container with id aa1acca30d71dd032ef8d368a4fe1456e0f18f4bd4e587d7ec00f7a41c0de9d6 Jul 16 12:45:51.478674 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.478651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865"] Jul 16 12:45:51.482364 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.482343 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-f4c78"] Jul 16 12:45:51.486518 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:51.486491 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cf452b_ac67_4fcb_bfa3_312e08ac1687.slice/crio-77d53b06c55772a5b196a02fb8b56a427486f862da73a36cd57c46ab4ceb9517 WatchSource:0}: Error finding container 77d53b06c55772a5b196a02fb8b56a427486f862da73a36cd57c46ab4ceb9517: Status 404 returned error can't find the container with id 77d53b06c55772a5b196a02fb8b56a427486f862da73a36cd57c46ab4ceb9517 Jul 16 12:45:51.487195 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:51.487171 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod192c0582_645d_451e_8f1a_edef1f89bbbd.slice/crio-08608be824c7ada1dbecb6fffb1920a3b6c432d6bdcabf3324c3359a77f80ff5 WatchSource:0}: Error finding container 08608be824c7ada1dbecb6fffb1920a3b6c432d6bdcabf3324c3359a77f80ff5: Status 404 returned error can't find the container with id 08608be824c7ada1dbecb6fffb1920a3b6c432d6bdcabf3324c3359a77f80ff5 Jul 16 12:45:51.488341 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:45:51.488324 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a27340_e369_4927_80d0_1f9b6cf7236d.slice/crio-eaeb82922709c61e6a07333db28fb114ae28d2ff0bc5c09bf830783383e7d3f6 WatchSource:0}: Error finding container eaeb82922709c61e6a07333db28fb114ae28d2ff0bc5c09bf830783383e7d3f6: Status 404 returned error can't find the container with id eaeb82922709c61e6a07333db28fb114ae28d2ff0bc5c09bf830783383e7d3f6 Jul 16 12:45:51.980315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.980276 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:45:51.983494 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:51.983463 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sbdj2\"" Jul 16 12:45:52.139382 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.139347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:52.139917 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:52.139560 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:52.139917 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:52.139624 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls podName:06467239-8bf8-4732-a273-e283e67af2dc nodeName:}" failed. No retries permitted until 2026-07-16 12:45:54.139605565 +0000 UTC m=+39.682212292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls") pod "dns-default-8l5pc" (UID: "06467239-8bf8-4732-a273-e283e67af2dc") : secret "dns-default-metrics-tls" not found Jul 16 12:45:52.161920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.161788 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" event={"ID":"192c0582-645d-451e-8f1a-edef1f89bbbd","Type":"ContainerStarted","Data":"08608be824c7ada1dbecb6fffb1920a3b6c432d6bdcabf3324c3359a77f80ff5"} Jul 16 12:45:52.164289 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.164234 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865" event={"ID":"75a27340-e369-4927-80d0-1f9b6cf7236d","Type":"ContainerStarted","Data":"eaeb82922709c61e6a07333db28fb114ae28d2ff0bc5c09bf830783383e7d3f6"} Jul 16 12:45:52.171573 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.170674 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2c635b3-06ef-492c-967e-f051ce3a388d" containerID="3056640c9fc8bd1078a29f4f1f7ea81bdf3da8a39f8e911f0afa4deb477875f4" exitCode=0 Jul 16 12:45:52.171573 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.170762 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" event={"ID":"d2c635b3-06ef-492c-967e-f051ce3a388d","Type":"ContainerDied","Data":"3056640c9fc8bd1078a29f4f1f7ea81bdf3da8a39f8e911f0afa4deb477875f4"} Jul 16 12:45:52.176050 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.176023 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492" event={"ID":"db1a9477-cd53-41f9-b36c-b9831895cca0","Type":"ContainerStarted","Data":"c8105d496715a1fd59c9dd444aca12d5df4e6f0bb63920e8158bafacb1e1c1d6"} Jul 16 12:45:52.179880 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.179858 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f4c78" event={"ID":"38cf452b-ac67-4fcb-bfa3-312e08ac1687","Type":"ContainerStarted","Data":"77d53b06c55772a5b196a02fb8b56a427486f862da73a36cd57c46ab4ceb9517"} Jul 16 12:45:52.182828 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.182789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-dswpk" event={"ID":"f1441a03-825b-4cb2-8d29-35bfd66d04f9","Type":"ContainerStarted","Data":"aa1acca30d71dd032ef8d368a4fe1456e0f18f4bd4e587d7ec00f7a41c0de9d6"} Jul 16 12:45:52.240209 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:52.240052 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:52.240360 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:52.240240 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:52.240360 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:52.240261 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fc699ffc-smbfr: secret "image-registry-tls" not found Jul 16 12:45:52.244366 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:52.244016 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls podName:2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:54.243995278 +0000 UTC m=+39.786602003 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls") pod "image-registry-58fc699ffc-smbfr" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71") : secret "image-registry-tls" not found Jul 16 12:45:53.191337 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:53.191195 2576 generic.go:356] "Generic (PLEG): container finished" podID="d2c635b3-06ef-492c-967e-f051ce3a388d" containerID="05e54eafd0bc50df791987ccbe415a0746caaa93e315720250551b7035d31a17" exitCode=0 Jul 16 12:45:53.191337 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:53.191272 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" event={"ID":"d2c635b3-06ef-492c-967e-f051ce3a388d","Type":"ContainerDied","Data":"05e54eafd0bc50df791987ccbe415a0746caaa93e315720250551b7035d31a17"} Jul 16 12:45:54.158096 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.158008 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:54.158268 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:54.158179 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:54.158268 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:54.158261 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls podName:06467239-8bf8-4732-a273-e283e67af2dc nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.158241619 +0000 UTC m=+43.700848348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls") pod "dns-default-8l5pc" (UID: "06467239-8bf8-4732-a273-e283e67af2dc") : secret "dns-default-metrics-tls" not found Jul 16 12:45:54.258894 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.258860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:54.259402 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:54.259048 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:54.259402 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:54.259072 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fc699ffc-smbfr: secret "image-registry-tls" not found Jul 16 12:45:54.259402 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:54.259147 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls podName:2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.25912564 +0000 UTC m=+43.801732368 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls") pod "image-registry-58fc699ffc-smbfr" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71") : secret "image-registry-tls" not found Jul 16 12:45:54.572802 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.572772 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gdrxd"] Jul 16 12:45:54.583735 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.583708 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:54.584207 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.584180 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gdrxd"] Jul 16 12:45:54.586963 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.586939 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-r5dpv\"" Jul 16 12:45:54.587095 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.586937 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 16 12:45:54.588191 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.588171 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 16 12:45:54.762124 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.762051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:54.762383 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.762274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5939f4ee-ca79-4129-9937-fd75490ec6f2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:54.863117 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.863042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5939f4ee-ca79-4129-9937-fd75490ec6f2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:54.863281 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.863116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:54.863350 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:54.863300 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:45:54.863455 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:54.863395 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert podName:5939f4ee-ca79-4129-9937-fd75490ec6f2 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:55.363371082 +0000 UTC m=+40.905977812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gdrxd" (UID: "5939f4ee-ca79-4129-9937-fd75490ec6f2") : secret "networking-console-plugin-cert" not found Jul 16 12:45:54.863841 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:54.863802 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5939f4ee-ca79-4129-9937-fd75490ec6f2-nginx-conf\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:55.222285 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.222257 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9s9qn"] Jul 16 12:45:55.229276 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.229253 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.232453 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.232423 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 16 12:45:55.232593 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.232450 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 16 12:45:55.232593 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.232460 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 16 12:45:55.232593 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.232455 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 16 12:45:55.233704 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.233687 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9fmr\"" Jul 16 12:45:55.249936 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.249854 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9s9qn"] Jul 16 12:45:55.368149 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.368111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.368719 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.368173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852279b4-b62e-4245-8859-e0baa4f102a0-crio-socket\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.368719 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.368224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852279b4-b62e-4245-8859-e0baa4f102a0-data-volume\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.368719 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.368358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.368719 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.368410 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:55.368719 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.368487 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9zsj\" (UniqueName: \"kubernetes.io/projected/852279b4-b62e-4245-8859-e0baa4f102a0-kube-api-access-n9zsj\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.368719 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.368516 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:45:55.368719 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.368589 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert podName:5939f4ee-ca79-4129-9937-fd75490ec6f2 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:56.368568949 +0000 UTC m=+41.911175692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gdrxd" (UID: "5939f4ee-ca79-4129-9937-fd75490ec6f2") : secret "networking-console-plugin-cert" not found Jul 16 12:45:55.469735 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.469653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.469735 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.469721 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852279b4-b62e-4245-8859-e0baa4f102a0-crio-socket\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.469984 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.469758 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852279b4-b62e-4245-8859-e0baa4f102a0-data-volume\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.469984 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.469776 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:45:55.469984 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.469832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.469984 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.469861 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:55.969842608 +0000 UTC m=+41.512449336 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : secret "insights-runtime-extractor-tls" not found Jul 16 12:45:55.469984 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.469934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9zsj\" (UniqueName: \"kubernetes.io/projected/852279b4-b62e-4245-8859-e0baa4f102a0-kube-api-access-n9zsj\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.469984 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.469963 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/852279b4-b62e-4245-8859-e0baa4f102a0-crio-socket\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.469984 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.469974 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:45:55.470320 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.470037 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:55.970026417 +0000 UTC m=+41.512633145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : configmap "kube-rbac-proxy" not found Jul 16 12:45:55.470320 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.470194 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/852279b4-b62e-4245-8859-e0baa4f102a0-data-volume\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.481869 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.481849 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9zsj\" (UniqueName: \"kubernetes.io/projected/852279b4-b62e-4245-8859-e0baa4f102a0-kube-api-access-n9zsj\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.975008 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.974964 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.975209 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:55.975094 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:55.975209 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.975163 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:45:55.975209 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.975192 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:45:55.975365 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.975246 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:56.975229811 +0000 UTC m=+42.517836556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : secret "insights-runtime-extractor-tls" not found Jul 16 12:45:55.975365 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:55.975265 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:56.975257176 +0000 UTC m=+42.517863902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : configmap "kube-rbac-proxy" not found Jul 16 12:45:56.380025 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:56.379949 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:56.380373 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:56.380090 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:45:56.380373 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:56.380153 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert podName:5939f4ee-ca79-4129-9937-fd75490ec6f2 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.380138269 +0000 UTC m=+43.922744994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gdrxd" (UID: "5939f4ee-ca79-4129-9937-fd75490ec6f2") : secret "networking-console-plugin-cert" not found Jul 16 12:45:56.888933 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:56.888877 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l764j_4642c698-57ae-49ad-a5f5-54c575f0fde9/dns-node-resolver/0.log" Jul 16 12:45:56.985567 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:56.985535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:56.985733 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:56.985642 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:56.985733 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:56.985660 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:45:56.985733 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:56.985725 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.985708272 +0000 UTC m=+44.528315008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : configmap "kube-rbac-proxy" not found Jul 16 12:45:56.985882 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:56.985777 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:45:56.985882 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:56.985855 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:45:58.98583763 +0000 UTC m=+44.528444361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : secret "insights-runtime-extractor-tls" not found Jul 16 12:45:57.208687 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.208650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" event={"ID":"192c0582-645d-451e-8f1a-edef1f89bbbd","Type":"ContainerStarted","Data":"8a7f81c9bb44bc7dff752936627ad558c779aaaf40502e56fd07cce4abcd2ff8"} Jul 16 12:45:57.210262 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.210229 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865" event={"ID":"75a27340-e369-4927-80d0-1f9b6cf7236d","Type":"ContainerStarted","Data":"0737d9127314ff75bdda22a13ace6f828bd5a974107e650bcddc1a3fa5c858c4"} Jul 16 12:45:57.213709 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.213689 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" event={"ID":"d2c635b3-06ef-492c-967e-f051ce3a388d","Type":"ContainerStarted","Data":"71a6e69b740c0a290689f9bbe11e2c78ddb3b94f0d63f314f3f482bfabeafea4"} Jul 16 12:45:57.215213 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.215187 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492" event={"ID":"db1a9477-cd53-41f9-b36c-b9831895cca0","Type":"ContainerStarted","Data":"e1b673d9df30b6b8492b546e8576ba3f8a06dc8ab292106f8cd8bcb720d124b6"} Jul 16 12:45:57.216623 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.216598 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-f4c78" event={"ID":"38cf452b-ac67-4fcb-bfa3-312e08ac1687","Type":"ContainerStarted","Data":"b56e8e910528ce9a0a12e73282c71f3abf10449149b920ec015582b9c013effa"} Jul 16 12:45:57.218054 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.218038 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/0.log" Jul 16 12:45:57.218153 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.218072 2576 generic.go:356] "Generic (PLEG): container finished" podID="f1441a03-825b-4cb2-8d29-35bfd66d04f9" containerID="413e69a23d327ecb18578acd8efffdc96719a43590676f7f346527559809e370" exitCode=255 Jul 16 12:45:57.218153 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.218093 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-dswpk" event={"ID":"f1441a03-825b-4cb2-8d29-35bfd66d04f9","Type":"ContainerDied","Data":"413e69a23d327ecb18578acd8efffdc96719a43590676f7f346527559809e370"} Jul 16 12:45:57.218268 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.218254 2576 scope.go:117] "RemoveContainer" containerID="413e69a23d327ecb18578acd8efffdc96719a43590676f7f346527559809e370" Jul 16 12:45:57.229984 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.229940 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" podStartSLOduration=2.105191076 podStartE2EDuration="7.229926257s" totalImagesPullingTime="5.124735181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:57.229444527 +0000 UTC m=+42.772051286" watchObservedRunningTime="2026-07-16 12:45:57.229926257 +0000 UTC m=+42.772533003" Jul 16 12:45:57.250436 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.250390 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-f4c78" podStartSLOduration=2.125169652 podStartE2EDuration="7.250371862s" totalImagesPullingTime="5.12520221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:57.249489459 +0000 UTC m=+42.792096206" watchObservedRunningTime="2026-07-16 12:45:57.250371862 +0000 UTC m=+42.792978705" Jul 16 12:45:57.278830 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.278769 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-gn492" podStartSLOduration=2.194057239 podStartE2EDuration="7.27875296s" totalImagesPullingTime="5.084695721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:57.277953374 +0000 UTC m=+42.820560122" watchObservedRunningTime="2026-07-16 12:45:57.27875296 +0000 UTC m=+42.821359729" Jul 16 12:45:57.302094 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.302046 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-zr865" podStartSLOduration=2.170291654 podStartE2EDuration="7.30202704s" totalImagesPullingTime="5.131735386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:57.301405838 +0000 UTC m=+42.844012584" watchObservedRunningTime="2026-07-16 12:45:57.30202704 +0000 UTC m=+42.844633789" Jul 16 12:45:57.329447 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:57.329403 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2kn5j" podStartSLOduration=11.17409862 podStartE2EDuration="42.329388202s" totalImagesPullingTime="31.155289582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:45:57.32703874 +0000 UTC m=+42.869645487" watchObservedRunningTime="2026-07-16 12:45:57.329388202 +0000 UTC m=+42.871994948" Jul 16 12:45:58.091708 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.091682 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztrvw_5664fccb-777b-4367-8239-458af04f6c29/node-ca/0.log" Jul 16 12:45:58.197576 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.197539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:45:58.197743 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:58.197640 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 16 12:45:58.197743 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:58.197700 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls podName:06467239-8bf8-4732-a273-e283e67af2dc nodeName:}" failed. No retries permitted until 2026-07-16 12:46:06.197686349 +0000 UTC m=+51.740293074 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls") pod "dns-default-8l5pc" (UID: "06467239-8bf8-4732-a273-e283e67af2dc") : secret "dns-default-metrics-tls" not found Jul 16 12:45:58.223199 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.223175 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 12:45:58.223569 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.223555 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/0.log" Jul 16 12:45:58.223623 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.223590 2576 generic.go:356] "Generic (PLEG): container finished" podID="f1441a03-825b-4cb2-8d29-35bfd66d04f9" containerID="9d6a4b12c2a03f6272b41b9f56bb0f8402c2b89f8ddb28be58ce1ac58715f43e" exitCode=255 Jul 16 12:45:58.223714 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.223692 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-dswpk" event={"ID":"f1441a03-825b-4cb2-8d29-35bfd66d04f9","Type":"ContainerDied","Data":"9d6a4b12c2a03f6272b41b9f56bb0f8402c2b89f8ddb28be58ce1ac58715f43e"} Jul 16 12:45:58.223763 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.223751 2576 scope.go:117] "RemoveContainer" containerID="413e69a23d327ecb18578acd8efffdc96719a43590676f7f346527559809e370" Jul 16 12:45:58.224020 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.223997 2576 scope.go:117] "RemoveContainer" containerID="9d6a4b12c2a03f6272b41b9f56bb0f8402c2b89f8ddb28be58ce1ac58715f43e" Jul 16 12:45:58.224184 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:58.224162 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-dswpk_openshift-console-operator(f1441a03-825b-4cb2-8d29-35bfd66d04f9)\"" pod="openshift-console-operator/console-operator-7866497494-dswpk" podUID="f1441a03-825b-4cb2-8d29-35bfd66d04f9" Jul 16 12:45:58.299865 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.299437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:45:58.299865 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:58.299635 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 16 12:45:58.299865 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:58.299651 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58fc699ffc-smbfr: secret "image-registry-tls" not found Jul 16 12:45:58.299865 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:58.299701 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls podName:2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:06.299682675 +0000 UTC m=+51.842289416 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls") pod "image-registry-58fc699ffc-smbfr" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71") : secret "image-registry-tls" not found Jul 16 12:45:58.400525 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.400453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:45:58.400636 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:58.400570 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:45:58.400636 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:58.400620 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert podName:5939f4ee-ca79-4129-9937-fd75490ec6f2 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:02.400608232 +0000 UTC m=+47.943214958 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gdrxd" (UID: "5939f4ee-ca79-4129-9937-fd75490ec6f2") : secret "networking-console-plugin-cert" not found Jul 16 12:45:58.688524 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:58.688487 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f4c78_38cf452b-ac67-4fcb-bfa3-312e08ac1687/serve-healthcheck-canary/0.log" Jul 16 12:45:59.006196 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:59.006124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:59.006196 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:59.006193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:45:59.006365 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:59.006246 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:45:59.006365 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:59.006281 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:45:59.006365 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:59.006303 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:03.006287867 +0000 UTC m=+48.548894595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : configmap "kube-rbac-proxy" not found Jul 16 12:45:59.006365 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:59.006330 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:03.006317417 +0000 UTC m=+48.548924142 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : secret "insights-runtime-extractor-tls" not found Jul 16 12:45:59.228170 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:59.228145 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 12:45:59.228519 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:45:59.228450 2576 scope.go:117] "RemoveContainer" containerID="9d6a4b12c2a03f6272b41b9f56bb0f8402c2b89f8ddb28be58ce1ac58715f43e" Jul 16 12:45:59.228606 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:45:59.228586 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-dswpk_openshift-console-operator(f1441a03-825b-4cb2-8d29-35bfd66d04f9)\"" pod="openshift-console-operator/console-operator-7866497494-dswpk" podUID="f1441a03-825b-4cb2-8d29-35bfd66d04f9" Jul 16 12:46:00.881140 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:00.881102 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:46:00.881493 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:00.881155 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:46:00.881493 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:00.881453 2576 scope.go:117] "RemoveContainer" containerID="9d6a4b12c2a03f6272b41b9f56bb0f8402c2b89f8ddb28be58ce1ac58715f43e" Jul 16 12:46:00.881604 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:00.881585 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-dswpk_openshift-console-operator(f1441a03-825b-4cb2-8d29-35bfd66d04f9)\"" pod="openshift-console-operator/console-operator-7866497494-dswpk" podUID="f1441a03-825b-4cb2-8d29-35bfd66d04f9" Jul 16 12:46:02.436639 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:02.436604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:46:02.437034 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:02.436760 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 16 12:46:02.437034 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:02.436848 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert podName:5939f4ee-ca79-4129-9937-fd75490ec6f2 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:10.436812203 +0000 UTC m=+55.979418939 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-gdrxd" (UID: "5939f4ee-ca79-4129-9937-fd75490ec6f2") : secret "networking-console-plugin-cert" not found Jul 16 12:46:03.041813 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:03.041772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:46:03.042002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:03.041895 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:46:03.042002 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:03.041899 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 16 12:46:03.042002 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:03.041965 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:11.041949944 +0000 UTC m=+56.584556669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : configmap "kube-rbac-proxy" not found Jul 16 12:46:03.042178 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:03.042026 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:46:03.042178 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:03.042082 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:11.042065941 +0000 UTC m=+56.584672672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : secret "insights-runtime-extractor-tls" not found Jul 16 12:46:06.266486 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:06.266452 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:46:06.268669 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:06.268643 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06467239-8bf8-4732-a273-e283e67af2dc-metrics-tls\") pod \"dns-default-8l5pc\" (UID: \"06467239-8bf8-4732-a273-e283e67af2dc\") " pod="openshift-dns/dns-default-8l5pc" Jul 16 12:46:06.304098 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:06.304075 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8l5pc" Jul 16 12:46:06.367399 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:06.367363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:46:06.370875 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:06.370844 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"image-registry-58fc699ffc-smbfr\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:46:06.408144 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:06.408116 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:46:06.459680 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:06.459652 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06467239_8bf8_4732_a273_e283e67af2dc.slice/crio-c591635f8d86a0912ffaaa01953b2510a090a8f547149a7413ffcce1d7696d6c WatchSource:0}: Error finding container c591635f8d86a0912ffaaa01953b2510a090a8f547149a7413ffcce1d7696d6c: Status 404 returned error can't find the container with id c591635f8d86a0912ffaaa01953b2510a090a8f547149a7413ffcce1d7696d6c Jul 16 12:46:06.461254 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:06.461232 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8l5pc"] Jul 16 12:46:06.565804 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:06.565731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58fc699ffc-smbfr"] Jul 16 12:46:06.567681 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:06.567654 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb5f5b5_bc11_495a_acf3_d3ba0bd37f71.slice/crio-f550b442ce9b33f1059e11d34deb05d070bebde2a90b686b196149a356407a68 WatchSource:0}: Error finding container f550b442ce9b33f1059e11d34deb05d070bebde2a90b686b196149a356407a68: Status 404 returned error can't find the container with id f550b442ce9b33f1059e11d34deb05d070bebde2a90b686b196149a356407a68 Jul 16 12:46:07.254951 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:07.254918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" event={"ID":"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71","Type":"ContainerStarted","Data":"45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a"} Jul 16 12:46:07.255133 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:07.254975 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" event={"ID":"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71","Type":"ContainerStarted","Data":"f550b442ce9b33f1059e11d34deb05d070bebde2a90b686b196149a356407a68"} Jul 16 12:46:07.255133 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:07.255010 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:46:07.256000 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:07.255973 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8l5pc" event={"ID":"06467239-8bf8-4732-a273-e283e67af2dc","Type":"ContainerStarted","Data":"c591635f8d86a0912ffaaa01953b2510a090a8f547149a7413ffcce1d7696d6c"} Jul 16 12:46:07.288402 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:07.288361 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" podStartSLOduration=17.288346718 podStartE2EDuration="17.288346718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:07.288109347 +0000 UTC m=+52.830716130" watchObservedRunningTime="2026-07-16 12:46:07.288346718 +0000 UTC m=+52.830953464" Jul 16 12:46:10.267248 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:10.267212 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8l5pc" event={"ID":"06467239-8bf8-4732-a273-e283e67af2dc","Type":"ContainerStarted","Data":"76212bc8c12c9dc09695b4ab72c8d35e64e77e15ae1bb3cb359148e853cc209f"} Jul 16 12:46:10.267670 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:10.267261 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8l5pc" event={"ID":"06467239-8bf8-4732-a273-e283e67af2dc","Type":"ContainerStarted","Data":"3f4a99a2a912a0ea4ef765fb5268e90726924cfc956c83161246a32b308210df"} Jul 16 12:46:10.267670 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:10.267336 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8l5pc" Jul 16 12:46:10.289655 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:10.289606 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8l5pc" podStartSLOduration=16.935605809 podStartE2EDuration="20.289591166s" totalImagesPullingTime="3.353985357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:10.289016287 +0000 UTC m=+55.831623035" watchObservedRunningTime="2026-07-16 12:46:10.289591166 +0000 UTC m=+55.832197916" Jul 16 12:46:10.504295 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:10.504265 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:46:10.506573 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:10.506554 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5939f4ee-ca79-4129-9937-fd75490ec6f2-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-gdrxd\" (UID: \"5939f4ee-ca79-4129-9937-fd75490ec6f2\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:46:10.792840 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:10.792778 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" Jul 16 12:46:10.918005 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:10.917966 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-gdrxd"] Jul 16 12:46:10.922626 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:10.922598 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5939f4ee_ca79_4129_9937_fd75490ec6f2.slice/crio-cbfc750616a0fa7d030e59d242bb8946f9a8c1b7d65b546dc8aaad8a490a1e11 WatchSource:0}: Error finding container cbfc750616a0fa7d030e59d242bb8946f9a8c1b7d65b546dc8aaad8a490a1e11: Status 404 returned error can't find the container with id cbfc750616a0fa7d030e59d242bb8946f9a8c1b7d65b546dc8aaad8a490a1e11 Jul 16 12:46:11.108567 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:11.108491 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:46:11.108567 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:11.108545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:46:11.108739 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:11.108631 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 16 12:46:11.108739 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:11.108680 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls podName:852279b4-b62e-4245-8859-e0baa4f102a0 nodeName:}" failed. No retries permitted until 2026-07-16 12:46:27.108666453 +0000 UTC m=+72.651273178 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9s9qn" (UID: "852279b4-b62e-4245-8859-e0baa4f102a0") : secret "insights-runtime-extractor-tls" not found Jul 16 12:46:11.109029 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:11.109010 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/852279b4-b62e-4245-8859-e0baa4f102a0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:46:11.270699 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:11.270660 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" event={"ID":"5939f4ee-ca79-4129-9937-fd75490ec6f2","Type":"ContainerStarted","Data":"cbfc750616a0fa7d030e59d242bb8946f9a8c1b7d65b546dc8aaad8a490a1e11"} Jul 16 12:46:12.275376 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:12.275294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" event={"ID":"5939f4ee-ca79-4129-9937-fd75490ec6f2","Type":"ContainerStarted","Data":"607070ba86164be81830ab857bb60529dc68b8b7c26c934b28264809797f8c8f"} Jul 16 12:46:12.296593 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:12.296546 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-gdrxd" podStartSLOduration=17.243954906 podStartE2EDuration="18.296531197s" totalImagesPullingTime="1.052576291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:12.295557489 +0000 UTC m=+57.838164237" watchObservedRunningTime="2026-07-16 12:46:12.296531197 +0000 UTC m=+57.839137943" Jul 16 12:46:14.146220 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:14.146192 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d62t6" Jul 16 12:46:15.980713 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:15.980682 2576 scope.go:117] "RemoveContainer" containerID="9d6a4b12c2a03f6272b41b9f56bb0f8402c2b89f8ddb28be58ce1ac58715f43e" Jul 16 12:46:16.293292 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:16.293222 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 12:46:16.293292 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:16.293279 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-dswpk" event={"ID":"f1441a03-825b-4cb2-8d29-35bfd66d04f9","Type":"ContainerStarted","Data":"ed4be8bb4351292093e0071414c3428f7fae9088b141a22b13766f708a79b723"} Jul 16 12:46:16.293570 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:16.293549 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:46:16.318197 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:16.318145 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-dswpk" podStartSLOduration=21.178575776 podStartE2EDuration="26.318131135s" totalImagesPullingTime="5.139555359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:16.317634281 +0000 UTC m=+61.860241028" watchObservedRunningTime="2026-07-16 12:46:16.318131135 +0000 UTC m=+61.860737859" Jul 16 12:46:17.016592 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.016564 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-dswpk" Jul 16 12:46:17.580330 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.580300 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh"] Jul 16 12:46:17.582904 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.582887 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.611305 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.611277 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 16 12:46:17.611426 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.611346 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 16 12:46:17.612416 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.612398 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 16 12:46:17.612508 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.612399 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 16 12:46:17.630303 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.630281 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58fc699ffc-smbfr"] Jul 16 12:46:17.631468 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.631449 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh"] Jul 16 12:46:17.662782 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.662760 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/283e67ec-d657-4d26-8793-2d06881a2de2-klusterlet-config\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.662883 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.662786 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h97pc\" (UniqueName: \"kubernetes.io/projected/283e67ec-d657-4d26-8793-2d06881a2de2-kube-api-access-h97pc\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.662883 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.662804 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/283e67ec-d657-4d26-8793-2d06881a2de2-tmp\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.680297 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.680274 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl"] Jul 16 12:46:17.682037 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.682023 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" Jul 16 12:46:17.686432 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.686413 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bp2xm\"" Jul 16 12:46:17.686629 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.686613 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 16 12:46:17.686971 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.686950 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 16 12:46:17.742173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.742140 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl"] Jul 16 12:46:17.758936 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.758913 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-69d9478f48-wlfkb"] Jul 16 12:46:17.760582 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.760569 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.763854 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.763836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/283e67ec-d657-4d26-8793-2d06881a2de2-klusterlet-config\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.763975 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.763859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h97pc\" (UniqueName: \"kubernetes.io/projected/283e67ec-d657-4d26-8793-2d06881a2de2-kube-api-access-h97pc\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.764041 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.763992 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/283e67ec-d657-4d26-8793-2d06881a2de2-tmp\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.764102 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.764094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtv4\" (UniqueName: \"kubernetes.io/projected/12c9c081-a9b6-4a29-9bb4-03943d237acd-kube-api-access-4jtv4\") pod \"migrator-7f59b54f6c-p9zjl\" (UID: \"12c9c081-a9b6-4a29-9bb4-03943d237acd\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" Jul 16 12:46:17.764399 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.764377 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/283e67ec-d657-4d26-8793-2d06881a2de2-tmp\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.766298 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.766279 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/283e67ec-d657-4d26-8793-2d06881a2de2-klusterlet-config\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.788308 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.788285 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69d9478f48-wlfkb"] Jul 16 12:46:17.791659 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.791636 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h97pc\" (UniqueName: \"kubernetes.io/projected/283e67ec-d657-4d26-8793-2d06881a2de2-kube-api-access-h97pc\") pod \"klusterlet-addon-workmgr-65849c5c9f-jfpbh\" (UID: \"283e67ec-d657-4d26-8793-2d06881a2de2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.864960 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.864889 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/210378ab-b60e-47ad-8751-32ad7feaff6f-installation-pull-secrets\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.864960 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.864919 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-registry-tls\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.864960 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.864962 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/210378ab-b60e-47ad-8751-32ad7feaff6f-registry-certificates\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.865161 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.864985 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-bound-sa-token\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.865161 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.865007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/210378ab-b60e-47ad-8751-32ad7feaff6f-ca-trust-extracted\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.865161 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.865055 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/210378ab-b60e-47ad-8751-32ad7feaff6f-image-registry-private-configuration\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.865161 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.865133 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtv4\" (UniqueName: \"kubernetes.io/projected/12c9c081-a9b6-4a29-9bb4-03943d237acd-kube-api-access-4jtv4\") pod \"migrator-7f59b54f6c-p9zjl\" (UID: \"12c9c081-a9b6-4a29-9bb4-03943d237acd\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" Jul 16 12:46:17.865295 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.865168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/210378ab-b60e-47ad-8751-32ad7feaff6f-trusted-ca\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.865295 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.865188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p88j\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-kube-api-access-6p88j\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.874957 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:17.874937 2576 projected.go:291] Couldn't get configMap openshift-kube-storage-version-migrator/openshift-service-ca.crt: configmap "openshift-service-ca.crt" not found Jul 16 12:46:17.874957 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:17.874955 2576 projected.go:196] Error preparing data for projected volume kube-api-access-4jtv4 for pod openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl: configmap "openshift-service-ca.crt" not found Jul 16 12:46:17.875085 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:17.874997 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/12c9c081-a9b6-4a29-9bb4-03943d237acd-kube-api-access-4jtv4 podName:12c9c081-a9b6-4a29-9bb4-03943d237acd nodeName:}" failed. No retries permitted until 2026-07-16 12:46:18.374982924 +0000 UTC m=+63.917589650 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4jtv4" (UniqueName: "kubernetes.io/projected/12c9c081-a9b6-4a29-9bb4-03943d237acd-kube-api-access-4jtv4") pod "migrator-7f59b54f6c-p9zjl" (UID: "12c9c081-a9b6-4a29-9bb4-03943d237acd") : configmap "openshift-service-ca.crt" not found Jul 16 12:46:17.891846 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.891807 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:17.966312 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.966147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/210378ab-b60e-47ad-8751-32ad7feaff6f-installation-pull-secrets\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.966312 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.966199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-registry-tls\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.966312 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.966249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/210378ab-b60e-47ad-8751-32ad7feaff6f-registry-certificates\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.966312 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.966278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-bound-sa-token\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.966312 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.966304 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/210378ab-b60e-47ad-8751-32ad7feaff6f-ca-trust-extracted\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.966608 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.966347 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/210378ab-b60e-47ad-8751-32ad7feaff6f-image-registry-private-configuration\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.966608 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.966431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/210378ab-b60e-47ad-8751-32ad7feaff6f-trusted-ca\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.966608 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.966464 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p88j\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-kube-api-access-6p88j\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.967144 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.967112 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/210378ab-b60e-47ad-8751-32ad7feaff6f-ca-trust-extracted\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.967688 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.967637 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/210378ab-b60e-47ad-8751-32ad7feaff6f-registry-certificates\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.968041 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.968017 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/210378ab-b60e-47ad-8751-32ad7feaff6f-trusted-ca\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.969100 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.969080 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/210378ab-b60e-47ad-8751-32ad7feaff6f-installation-pull-secrets\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.970196 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.970179 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-registry-tls\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.970422 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.970406 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/210378ab-b60e-47ad-8751-32ad7feaff6f-image-registry-private-configuration\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.988847 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.988093 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-bound-sa-token\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:17.990308 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:17.990284 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p88j\" (UniqueName: \"kubernetes.io/projected/210378ab-b60e-47ad-8751-32ad7feaff6f-kube-api-access-6p88j\") pod \"image-registry-69d9478f48-wlfkb\" (UID: \"210378ab-b60e-47ad-8751-32ad7feaff6f\") " pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:18.022626 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.022600 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh"] Jul 16 12:46:18.025483 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:18.025461 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283e67ec_d657_4d26_8793_2d06881a2de2.slice/crio-475899eecdd6d9a7bbeccc153bff2ffffde85fa937c6126113edc1e545d96d1d WatchSource:0}: Error finding container 475899eecdd6d9a7bbeccc153bff2ffffde85fa937c6126113edc1e545d96d1d: Status 404 returned error can't find the container with id 475899eecdd6d9a7bbeccc153bff2ffffde85fa937c6126113edc1e545d96d1d Jul 16 12:46:18.074196 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.074168 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:18.220771 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.220747 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-69d9478f48-wlfkb"] Jul 16 12:46:18.222973 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:18.222948 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod210378ab_b60e_47ad_8751_32ad7feaff6f.slice/crio-969f062e37b7c5f1761fd6f44a847bcf6573c1120ff2f7dcb97664e42df8b5b2 WatchSource:0}: Error finding container 969f062e37b7c5f1761fd6f44a847bcf6573c1120ff2f7dcb97664e42df8b5b2: Status 404 returned error can't find the container with id 969f062e37b7c5f1761fd6f44a847bcf6573c1120ff2f7dcb97664e42df8b5b2 Jul 16 12:46:18.300994 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.300948 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" event={"ID":"283e67ec-d657-4d26-8793-2d06881a2de2","Type":"ContainerStarted","Data":"475899eecdd6d9a7bbeccc153bff2ffffde85fa937c6126113edc1e545d96d1d"} Jul 16 12:46:18.302646 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.302620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" event={"ID":"210378ab-b60e-47ad-8751-32ad7feaff6f","Type":"ContainerStarted","Data":"a82bc2b27135892f9b2db7b5a87f7da17a351f4929958f54eae05efdc740a5ca"} Jul 16 12:46:18.302762 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.302662 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" event={"ID":"210378ab-b60e-47ad-8751-32ad7feaff6f","Type":"ContainerStarted","Data":"969f062e37b7c5f1761fd6f44a847bcf6573c1120ff2f7dcb97664e42df8b5b2"} Jul 16 12:46:18.337951 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.337904 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" podStartSLOduration=1.337887298 podStartE2EDuration="1.337887298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:18.33757849 +0000 UTC m=+63.880185239" watchObservedRunningTime="2026-07-16 12:46:18.337887298 +0000 UTC m=+63.880494044" Jul 16 12:46:18.471373 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.471335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtv4\" (UniqueName: \"kubernetes.io/projected/12c9c081-a9b6-4a29-9bb4-03943d237acd-kube-api-access-4jtv4\") pod \"migrator-7f59b54f6c-p9zjl\" (UID: \"12c9c081-a9b6-4a29-9bb4-03943d237acd\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" Jul 16 12:46:18.473541 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.473520 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtv4\" (UniqueName: \"kubernetes.io/projected/12c9c081-a9b6-4a29-9bb4-03943d237acd-kube-api-access-4jtv4\") pod \"migrator-7f59b54f6c-p9zjl\" (UID: \"12c9c081-a9b6-4a29-9bb4-03943d237acd\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" Jul 16 12:46:18.590658 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.590621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" Jul 16 12:46:18.716395 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:18.716344 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl"] Jul 16 12:46:18.719220 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:18.719192 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c9c081_a9b6_4a29_9bb4_03943d237acd.slice/crio-070f1907808a17d44bef81305f34b157dc1347f8d5a2794d50086ff61e67f223 WatchSource:0}: Error finding container 070f1907808a17d44bef81305f34b157dc1347f8d5a2794d50086ff61e67f223: Status 404 returned error can't find the container with id 070f1907808a17d44bef81305f34b157dc1347f8d5a2794d50086ff61e67f223 Jul 16 12:46:19.308259 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:19.308218 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" event={"ID":"12c9c081-a9b6-4a29-9bb4-03943d237acd","Type":"ContainerStarted","Data":"070f1907808a17d44bef81305f34b157dc1347f8d5a2794d50086ff61e67f223"} Jul 16 12:46:19.308743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:19.308494 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:20.273231 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.273202 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8l5pc" Jul 16 12:46:20.694471 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.694437 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:46:20.697741 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.697715 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 16 12:46:20.707905 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.707877 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b6659fe-fabb-4a30-aaba-3615b0e73f11-original-pull-secret\") pod \"global-pull-secret-syncer-pvcrz\" (UID: \"0b6659fe-fabb-4a30-aaba-3615b0e73f11\") " pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:46:20.795556 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.795517 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:46:20.795734 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.795621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:46:20.798414 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.798395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scjvm\" (UniqueName: \"kubernetes.io/projected/ad46a259-f207-45c2-8379-5040cab7d186-kube-api-access-scjvm\") pod \"network-check-target-kmxqh\" (UID: \"ad46a259-f207-45c2-8379-5040cab7d186\") " pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:46:20.799044 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.799026 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 16 12:46:20.808474 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.808453 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cf65300-74d2-4492-8245-b3984de67048-metrics-certs\") pod \"network-metrics-daemon-hpwb2\" (UID: \"5cf65300-74d2-4492-8245-b3984de67048\") " pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:46:20.995537 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.995456 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xrlr5\"" Jul 16 12:46:20.997521 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:20.997500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-pvcrz" Jul 16 12:46:21.002524 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:21.002495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hpwb2" Jul 16 12:46:21.096868 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:21.096834 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sbdj2\"" Jul 16 12:46:21.104364 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:21.104339 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:46:21.895392 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:21.895321 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kmxqh"] Jul 16 12:46:21.897874 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:21.897851 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad46a259_f207_45c2_8379_5040cab7d186.slice/crio-9ce1918656ad44c96939cae3971a3d24fb49f2aa685e218cda07c3993af5cab4 WatchSource:0}: Error finding container 9ce1918656ad44c96939cae3971a3d24fb49f2aa685e218cda07c3993af5cab4: Status 404 returned error can't find the container with id 9ce1918656ad44c96939cae3971a3d24fb49f2aa685e218cda07c3993af5cab4 Jul 16 12:46:22.118616 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.118543 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hpwb2"] Jul 16 12:46:22.121795 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.121765 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-pvcrz"] Jul 16 12:46:22.122449 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:22.122424 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf65300_74d2_4492_8245_b3984de67048.slice/crio-041c274b7feededac1d5daaee75898a01d0c715240782379d22019da0de0d474 WatchSource:0}: Error finding container 041c274b7feededac1d5daaee75898a01d0c715240782379d22019da0de0d474: Status 404 returned error can't find the container with id 041c274b7feededac1d5daaee75898a01d0c715240782379d22019da0de0d474 Jul 16 12:46:22.124827 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:22.124792 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6659fe_fabb_4a30_aaba_3615b0e73f11.slice/crio-02cb030fd470e45b09b832e38cffa35e780e35623010f7a7b5b3ac9e8f3e4b91 WatchSource:0}: Error finding container 02cb030fd470e45b09b832e38cffa35e780e35623010f7a7b5b3ac9e8f3e4b91: Status 404 returned error can't find the container with id 02cb030fd470e45b09b832e38cffa35e780e35623010f7a7b5b3ac9e8f3e4b91 Jul 16 12:46:22.326388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.326351 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" event={"ID":"12c9c081-a9b6-4a29-9bb4-03943d237acd","Type":"ContainerStarted","Data":"433b9b76c853719285320e0692944c7b6ef561878a02182e4f77cd231e5a2373"} Jul 16 12:46:22.326557 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.326405 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" event={"ID":"12c9c081-a9b6-4a29-9bb4-03943d237acd","Type":"ContainerStarted","Data":"072c415cae3bd720ed5cf1e0b2216b384222a176d5ac65aa4d6adaad20628953"} Jul 16 12:46:22.327701 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.327680 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kmxqh" event={"ID":"ad46a259-f207-45c2-8379-5040cab7d186","Type":"ContainerStarted","Data":"22f518f6744728f8938e906912747b03c7e7e85e844d82933458115fdc4a5047"} Jul 16 12:46:22.327840 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.327716 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kmxqh" event={"ID":"ad46a259-f207-45c2-8379-5040cab7d186","Type":"ContainerStarted","Data":"9ce1918656ad44c96939cae3971a3d24fb49f2aa685e218cda07c3993af5cab4"} Jul 16 12:46:22.327840 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.327733 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:46:22.329005 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.328977 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" event={"ID":"283e67ec-d657-4d26-8793-2d06881a2de2","Type":"ContainerStarted","Data":"8db7c7aecbf83f53d91c97293c12ea2b2e089c3386d1ee74b34b1de08f9b8304"} Jul 16 12:46:22.329123 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.329100 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:22.330292 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.330268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pvcrz" event={"ID":"0b6659fe-fabb-4a30-aaba-3615b0e73f11","Type":"ContainerStarted","Data":"02cb030fd470e45b09b832e38cffa35e780e35623010f7a7b5b3ac9e8f3e4b91"} Jul 16 12:46:22.331038 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.331021 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" Jul 16 12:46:22.331544 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.331522 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpwb2" event={"ID":"5cf65300-74d2-4492-8245-b3984de67048","Type":"ContainerStarted","Data":"041c274b7feededac1d5daaee75898a01d0c715240782379d22019da0de0d474"} Jul 16 12:46:22.349581 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.349543 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-p9zjl" podStartSLOduration=2.332852254 podStartE2EDuration="5.349531648s" totalImagesPullingTime="3.016679394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:22.348651937 +0000 UTC m=+67.891258685" watchObservedRunningTime="2026-07-16 12:46:22.349531648 +0000 UTC m=+67.892138372" Jul 16 12:46:22.380233 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.380160 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-65849c5c9f-jfpbh" podStartSLOduration=1.590934607 podStartE2EDuration="5.380148775s" totalImagesPullingTime="3.789214168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:22.379435251 +0000 UTC m=+67.922042010" watchObservedRunningTime="2026-07-16 12:46:22.380148775 +0000 UTC m=+67.922755522" Jul 16 12:46:22.405135 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:22.405095 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kmxqh" podStartSLOduration=67.405083237 podStartE2EDuration="1m7.405083237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:22.403585414 +0000 UTC m=+67.946192164" watchObservedRunningTime="2026-07-16 12:46:22.405083237 +0000 UTC m=+67.947689984" Jul 16 12:46:24.341516 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:24.341483 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpwb2" event={"ID":"5cf65300-74d2-4492-8245-b3984de67048","Type":"ContainerStarted","Data":"815b17b4f00e79be8fe0916b3700ff777467f9c488f31559fcf475eebf726774"} Jul 16 12:46:24.341965 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:24.341536 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hpwb2" event={"ID":"5cf65300-74d2-4492-8245-b3984de67048","Type":"ContainerStarted","Data":"f74e66a6dfb5dc58f0fe493061f334e298dde0a9efacac56642246d1af217b49"} Jul 16 12:46:24.361557 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:24.361503 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hpwb2" podStartSLOduration=67.877830288 podStartE2EDuration="1m9.361482644s" totalImagesPullingTime="1.483652356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:24.36042839 +0000 UTC m=+69.903035148" watchObservedRunningTime="2026-07-16 12:46:24.361482644 +0000 UTC m=+69.904089392" Jul 16 12:46:27.149917 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:27.149880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:46:27.152168 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:27.152138 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/852279b4-b62e-4245-8859-e0baa4f102a0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9s9qn\" (UID: \"852279b4-b62e-4245-8859-e0baa4f102a0\") " pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:46:27.343324 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:27.343290 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g9fmr\"" Jul 16 12:46:27.350518 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:27.350492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9s9qn" Jul 16 12:46:27.353832 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:27.353793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-pvcrz" event={"ID":"0b6659fe-fabb-4a30-aaba-3615b0e73f11","Type":"ContainerStarted","Data":"0ea5a504b2ee1a03bef8b698497d166e593644c9db50161c98215f2c26dddc21"} Jul 16 12:46:27.372096 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:27.372053 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-pvcrz" podStartSLOduration=67.889189357 podStartE2EDuration="1m12.372038793s" totalImagesPullingTime="4.482849436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:27.371361818 +0000 UTC m=+72.913968566" watchObservedRunningTime="2026-07-16 12:46:27.372038793 +0000 UTC m=+72.914645540" Jul 16 12:46:27.492427 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:27.492402 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9s9qn"] Jul 16 12:46:27.495376 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:27.495348 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852279b4_b62e_4245_8859_e0baa4f102a0.slice/crio-24cdd1b078e358e378861d42803d21022a32230b7b24ac783a9f83dfbfaf12cb WatchSource:0}: Error finding container 24cdd1b078e358e378861d42803d21022a32230b7b24ac783a9f83dfbfaf12cb: Status 404 returned error can't find the container with id 24cdd1b078e358e378861d42803d21022a32230b7b24ac783a9f83dfbfaf12cb Jul 16 12:46:27.635611 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:27.635584 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:46:28.364190 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:28.364077 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9s9qn" event={"ID":"852279b4-b62e-4245-8859-e0baa4f102a0","Type":"ContainerStarted","Data":"24cdd1b078e358e378861d42803d21022a32230b7b24ac783a9f83dfbfaf12cb"} Jul 16 12:46:33.384892 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:33.384807 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9s9qn" event={"ID":"852279b4-b62e-4245-8859-e0baa4f102a0","Type":"ContainerStarted","Data":"d3a976b99ac7eb6ba949a5eba564c58f1074d4fd649616a59cb85a41d110dd9a"} Jul 16 12:46:36.399523 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:36.399481 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9s9qn" event={"ID":"852279b4-b62e-4245-8859-e0baa4f102a0","Type":"ContainerStarted","Data":"1737872d76c5e5f6abf399615193a4eac5bac5a159c29cd8329c947891e237cf"} Jul 16 12:46:37.146919 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.146869 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7454ffd75d-kwpms"] Jul 16 12:46:37.149571 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.149544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.152433 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.152411 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 16 12:46:37.152566 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.152430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-678j9\"" Jul 16 12:46:37.154217 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.153895 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 16 12:46:37.154217 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.153918 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 16 12:46:37.154217 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.153963 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 16 12:46:37.154217 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.153921 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 16 12:46:37.154217 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.154110 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 16 12:46:37.154217 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.154122 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 16 12:46:37.159758 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.159732 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 16 12:46:37.163149 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.163088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7454ffd75d-kwpms"] Jul 16 12:46:37.241600 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.241566 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-oauth-config\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.241836 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.241642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-serving-cert\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.241836 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.241674 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-service-ca\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.241836 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.241700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-trusted-ca-bundle\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.241836 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.241736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxm64\" (UniqueName: \"kubernetes.io/projected/5c621b5c-23ac-41a6-b895-b7974b08d8f4-kube-api-access-cxm64\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.241836 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.241771 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-config\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.241836 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.241800 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-oauth-serving-cert\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.342654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.342612 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-oauth-config\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.342851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.342677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-serving-cert\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.342851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.342703 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-service-ca\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.342851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.342723 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-trusted-ca-bundle\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.342851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.342752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxm64\" (UniqueName: \"kubernetes.io/projected/5c621b5c-23ac-41a6-b895-b7974b08d8f4-kube-api-access-cxm64\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.342851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.342781 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-config\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.343188 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.343164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-oauth-serving-cert\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.343618 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.343508 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-service-ca\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.343756 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.343725 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-config\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.343897 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.343855 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-oauth-serving-cert\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.344002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.343979 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-trusted-ca-bundle\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.345487 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.345459 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-oauth-config\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.346082 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.346059 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-serving-cert\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.355096 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.355073 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxm64\" (UniqueName: \"kubernetes.io/projected/5c621b5c-23ac-41a6-b895-b7974b08d8f4-kube-api-access-cxm64\") pod \"console-7454ffd75d-kwpms\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.460924 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.460894 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:37.605106 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:37.605079 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7454ffd75d-kwpms"] Jul 16 12:46:39.243283 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:39.243236 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c621b5c_23ac_41a6_b895_b7974b08d8f4.slice/crio-bb13cb9059109036e3ec8b8668d74e5aa8e6d66a3280bbd062f53bcddc33f0f7 WatchSource:0}: Error finding container bb13cb9059109036e3ec8b8668d74e5aa8e6d66a3280bbd062f53bcddc33f0f7: Status 404 returned error can't find the container with id bb13cb9059109036e3ec8b8668d74e5aa8e6d66a3280bbd062f53bcddc33f0f7 Jul 16 12:46:39.413929 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:39.413893 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7454ffd75d-kwpms" event={"ID":"5c621b5c-23ac-41a6-b895-b7974b08d8f4","Type":"ContainerStarted","Data":"bb13cb9059109036e3ec8b8668d74e5aa8e6d66a3280bbd062f53bcddc33f0f7"} Jul 16 12:46:39.416148 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:39.416120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9s9qn" event={"ID":"852279b4-b62e-4245-8859-e0baa4f102a0","Type":"ContainerStarted","Data":"85f65abae8e214ccab90edd3979b6ca13e05562c2c290f3f461ed2d5dbd5afd3"} Jul 16 12:46:39.439450 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:39.439406 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9s9qn" podStartSLOduration=32.783646264 podStartE2EDuration="44.439388776s" totalImagesPullingTime="11.655742512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 12:45:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:39.438192182 +0000 UTC m=+84.980798940" watchObservedRunningTime="2026-07-16 12:46:39.439388776 +0000 UTC m=+84.981995520" Jul 16 12:46:40.318665 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:40.318634 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-69d9478f48-wlfkb" Jul 16 12:46:42.428346 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.428313 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7454ffd75d-kwpms" event={"ID":"5c621b5c-23ac-41a6-b895-b7974b08d8f4","Type":"ContainerStarted","Data":"ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb"} Jul 16 12:46:42.451727 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.451684 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7454ffd75d-kwpms" podStartSLOduration=2.536145295 podStartE2EDuration="5.451668201s" totalImagesPullingTime="2.915522906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:42.45040412 +0000 UTC m=+87.993010886" watchObservedRunningTime="2026-07-16 12:46:42.451668201 +0000 UTC m=+87.994274949" Jul 16 12:46:42.650571 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.650477 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" podUID="2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" containerName="registry" containerID="cri-o://45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a" gracePeriod=30 Jul 16 12:46:42.886241 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.886216 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:46:42.984637 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.984612 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwhgm\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-kube-api-access-vwhgm\") pod \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " Jul 16 12:46:42.984771 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.984645 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-certificates\") pod \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " Jul 16 12:46:42.984771 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.984689 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") pod \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " Jul 16 12:46:42.984895 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.984792 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-ca-trust-extracted\") pod \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " Jul 16 12:46:42.984895 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.984880 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-bound-sa-token\") pod \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " Jul 16 12:46:42.985010 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.984926 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-image-registry-private-configuration\") pod \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " Jul 16 12:46:42.985010 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.984967 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-trusted-ca\") pod \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " Jul 16 12:46:42.985108 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.985009 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-installation-pull-secrets\") pod \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\" (UID: \"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71\") " Jul 16 12:46:42.985108 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.985016 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:46:42.985579 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.985538 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:46:42.987334 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.987289 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:46:42.987459 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.987431 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-kube-api-access-vwhgm" (OuterVolumeSpecName: "kube-api-access-vwhgm") pod "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71"). InnerVolumeSpecName "kube-api-access-vwhgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:46:42.987634 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.987610 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:46:42.987726 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.987675 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:46:42.987726 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.987695 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:46:42.993922 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:42.993804 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" (UID: "2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:46:43.085730 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.085667 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-trusted-ca\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:46:43.085730 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.085689 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-installation-pull-secrets\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:46:43.085730 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.085699 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhgm\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-kube-api-access-vwhgm\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:46:43.085730 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.085709 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-certificates\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:46:43.085730 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.085717 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-registry-tls\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:46:43.085730 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.085726 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-ca-trust-extracted\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:46:43.085730 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.085734 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-bound-sa-token\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:46:43.086036 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.085743 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71-image-registry-private-configuration\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:46:43.433182 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.433145 2576 generic.go:356] "Generic (PLEG): container finished" podID="2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" containerID="45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a" exitCode=0 Jul 16 12:46:43.433544 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.433195 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" Jul 16 12:46:43.433544 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.433238 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" event={"ID":"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71","Type":"ContainerDied","Data":"45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a"} Jul 16 12:46:43.433544 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.433295 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-58fc699ffc-smbfr" event={"ID":"2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71","Type":"ContainerDied","Data":"f550b442ce9b33f1059e11d34deb05d070bebde2a90b686b196149a356407a68"} Jul 16 12:46:43.433544 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.433317 2576 scope.go:117] "RemoveContainer" containerID="45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a" Jul 16 12:46:43.441229 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.441210 2576 scope.go:117] "RemoveContainer" containerID="45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a" Jul 16 12:46:43.441505 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:43.441487 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a\": container with ID starting with 45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a not found: ID does not exist" containerID="45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a" Jul 16 12:46:43.441551 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.441514 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a"} err="failed to get container status \"45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a\": rpc error: code = NotFound desc = could not find container \"45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a\": container with ID starting with 45a1fdeab9858ce127a52ea6e742e1f0aa644dfcd6c00fc2ebf775f46172292a not found: ID does not exist" Jul 16 12:46:43.456104 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.456078 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58fc699ffc-smbfr"] Jul 16 12:46:43.461136 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:43.461116 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58fc699ffc-smbfr"] Jul 16 12:46:44.988195 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:44.988161 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" path="/var/lib/kubelet/pods/2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71/volumes" Jul 16 12:46:47.461724 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:47.461693 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:47.462180 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:47.461754 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:47.466698 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:47.466676 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:48.159561 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.159530 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-2dtkn"] Jul 16 12:46:48.159837 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.159807 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" containerName="registry" Jul 16 12:46:48.159837 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.159833 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" containerName="registry" Jul 16 12:46:48.159927 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.159897 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb5f5b5-bc11-495a-acf3-d3ba0bd37f71" containerName="registry" Jul 16 12:46:48.167609 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.167590 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.170510 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.170489 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 16 12:46:48.171925 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.171904 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 16 12:46:48.172103 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.172083 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 16 12:46:48.172206 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.171974 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 16 12:46:48.172206 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.172086 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 16 12:46:48.172329 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.171952 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-f2szm\"" Jul 16 12:46:48.172329 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.172100 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 16 12:46:48.173370 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.173353 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hnb6q"] Jul 16 12:46:48.176738 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.176714 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-2dtkn"] Jul 16 12:46:48.176851 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.176792 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.179539 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.179522 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 16 12:46:48.179539 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.179531 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2nc5p\"" Jul 16 12:46:48.182225 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.182198 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 16 12:46:48.182448 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.182433 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 16 12:46:48.219309 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-root\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.219454 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-textfile\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.219454 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.219454 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219363 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q699\" (UniqueName: \"kubernetes.io/projected/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-api-access-4q699\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.219454 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-sys\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.219598 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.219598 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219482 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.219598 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219502 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7325a195-047d-44d5-86e0-31c918f6a9ea-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.219598 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-wtmp\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.219598 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7325a195-047d-44d5-86e0-31c918f6a9ea-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.219756 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219612 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5pf4\" (UniqueName: \"kubernetes.io/projected/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-kube-api-access-w5pf4\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.219756 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219645 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.219756 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219664 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-metrics-client-ca\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.219756 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.219756 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.219724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-tls\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320259 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320217 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q699\" (UniqueName: \"kubernetes.io/projected/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-api-access-4q699\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.320259 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-sys\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320473 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320290 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.320473 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-sys\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320473 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.320473 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7325a195-047d-44d5-86e0-31c918f6a9ea-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.320473 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-wtmp\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320473 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320467 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7325a195-047d-44d5-86e0-31c918f6a9ea-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320498 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5pf4\" (UniqueName: \"kubernetes.io/projected/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-kube-api-access-w5pf4\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-metrics-client-ca\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-tls\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320616 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-wtmp\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320650 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-root\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-textfile\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.320785 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320708 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.321267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.320905 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7325a195-047d-44d5-86e0-31c918f6a9ea-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.321267 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:48.320986 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 16 12:46:48.321267 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:46:48.321046 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-tls podName:f5b36e44-0e72-4ec2-b640-ab003f3dfa6a nodeName:}" failed. No retries permitted until 2026-07-16 12:46:48.821026202 +0000 UTC m=+94.363632927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-tls") pod "node-exporter-hnb6q" (UID: "f5b36e44-0e72-4ec2-b640-ab003f3dfa6a") : secret "node-exporter-tls" not found Jul 16 12:46:48.321267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.321130 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.321267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.321203 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7325a195-047d-44d5-86e0-31c918f6a9ea-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.321267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.321233 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-root\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.321267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.321238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-metrics-client-ca\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.321614 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.321475 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-textfile\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.321676 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.321665 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-accelerators-collector-config\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.323063 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.323038 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.323176 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.323095 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.323176 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.323114 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.330764 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.330739 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5pf4\" (UniqueName: \"kubernetes.io/projected/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-kube-api-access-w5pf4\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.330868 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.330768 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q699\" (UniqueName: \"kubernetes.io/projected/7325a195-047d-44d5-86e0-31c918f6a9ea-kube-api-access-4q699\") pod \"kube-state-metrics-785778ff74-2dtkn\" (UID: \"7325a195-047d-44d5-86e0-31c918f6a9ea\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.455966 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.455941 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:46:48.477382 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.477357 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" Jul 16 12:46:48.616644 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.616613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-2dtkn"] Jul 16 12:46:48.620038 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:48.620012 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7325a195_047d_44d5_86e0_31c918f6a9ea.slice/crio-734bb1d64f636d8651533dd5699d360be8620a216b5671ed20e7e6bba62eb98d WatchSource:0}: Error finding container 734bb1d64f636d8651533dd5699d360be8620a216b5671ed20e7e6bba62eb98d: Status 404 returned error can't find the container with id 734bb1d64f636d8651533dd5699d360be8620a216b5671ed20e7e6bba62eb98d Jul 16 12:46:48.825528 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.825446 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-tls\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:48.827632 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:48.827615 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f5b36e44-0e72-4ec2-b640-ab003f3dfa6a-node-exporter-tls\") pod \"node-exporter-hnb6q\" (UID: \"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a\") " pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:49.086843 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:49.086745 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hnb6q" Jul 16 12:46:49.098328 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:49.098301 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b36e44_0e72_4ec2_b640_ab003f3dfa6a.slice/crio-fbadf982aec819bb2ca372ef81e8fda36d04eeb33d5fcab137807e08d2363c56 WatchSource:0}: Error finding container fbadf982aec819bb2ca372ef81e8fda36d04eeb33d5fcab137807e08d2363c56: Status 404 returned error can't find the container with id fbadf982aec819bb2ca372ef81e8fda36d04eeb33d5fcab137807e08d2363c56 Jul 16 12:46:49.456446 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:49.456407 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnb6q" event={"ID":"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a","Type":"ContainerStarted","Data":"fbadf982aec819bb2ca372ef81e8fda36d04eeb33d5fcab137807e08d2363c56"} Jul 16 12:46:49.457548 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:49.457524 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" event={"ID":"7325a195-047d-44d5-86e0-31c918f6a9ea","Type":"ContainerStarted","Data":"734bb1d64f636d8651533dd5699d360be8620a216b5671ed20e7e6bba62eb98d"} Jul 16 12:46:50.463404 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:50.463366 2576 generic.go:356] "Generic (PLEG): container finished" podID="f5b36e44-0e72-4ec2-b640-ab003f3dfa6a" containerID="e055603bfd47dab560154f61ac998a4f1782fc3379605b328fe715f387ceac5f" exitCode=0 Jul 16 12:46:50.463881 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:50.463415 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnb6q" event={"ID":"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a","Type":"ContainerDied","Data":"e055603bfd47dab560154f61ac998a4f1782fc3379605b328fe715f387ceac5f"} Jul 16 12:46:51.469773 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:51.469736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnb6q" event={"ID":"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a","Type":"ContainerStarted","Data":"ac71071cc4a55aaa8d945bd5227da1af8aa2030eef49b21ae41a81ea71be1422"} Jul 16 12:46:51.470211 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:51.469795 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hnb6q" event={"ID":"f5b36e44-0e72-4ec2-b640-ab003f3dfa6a","Type":"ContainerStarted","Data":"3ffdeedf17129dd11177f59c62e99a29aab843bb68c23ab78ce684ae9e3b5984"} Jul 16 12:46:51.471906 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:51.471882 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" event={"ID":"7325a195-047d-44d5-86e0-31c918f6a9ea","Type":"ContainerStarted","Data":"7e2726ffce190d9cfdba598e54a3033099bb018b69a45963cbeeec55b6d7971c"} Jul 16 12:46:51.472030 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:51.471939 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" event={"ID":"7325a195-047d-44d5-86e0-31c918f6a9ea","Type":"ContainerStarted","Data":"6c1d4230c74d769955bc899f98dd3ae85aa3e538bdf6866d57b854d6c3f50cda"} Jul 16 12:46:51.472030 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:51.471971 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" event={"ID":"7325a195-047d-44d5-86e0-31c918f6a9ea","Type":"ContainerStarted","Data":"da948c7cd5c45758e3e728c059d1709089b2208ce7761d22673bb1f7aa032a5f"} Jul 16 12:46:51.492756 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:51.492707 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hnb6q" podStartSLOduration=2.473271016 podStartE2EDuration="3.492690928s" totalImagesPullingTime="954.387656ms" totalInitContainerRuntime="65.032256ms" isStatefulPod=true podCreationTimestamp="2026-07-16 12:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:51.492110178 +0000 UTC m=+97.034716925" watchObservedRunningTime="2026-07-16 12:46:51.492690928 +0000 UTC m=+97.035297675" Jul 16 12:46:51.517792 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:51.517745 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-2dtkn" podStartSLOduration=1.133300655 podStartE2EDuration="3.517730482s" totalImagesPullingTime="2.384429827s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:51.517150596 +0000 UTC m=+97.059757356" watchObservedRunningTime="2026-07-16 12:46:51.517730482 +0000 UTC m=+97.060337259" Jul 16 12:46:53.241184 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.241150 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-557ff67dcd-znxkl"] Jul 16 12:46:53.272374 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.272341 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-557ff67dcd-znxkl"] Jul 16 12:46:53.272505 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.272452 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.275408 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.275383 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 16 12:46:53.275550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.275426 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 16 12:46:53.275550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.275434 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 16 12:46:53.275550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.275451 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-tzctm\"" Jul 16 12:46:53.275550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.275431 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 16 12:46:53.275550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.275427 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 16 12:46:53.280695 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.280671 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 16 12:46:53.337805 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.337783 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kmxqh" Jul 16 12:46:53.363619 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.363589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-secret-telemeter-client\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.363767 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.363626 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.363767 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.363702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-telemeter-trusted-ca-bundle\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.363767 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.363751 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-serving-certs-ca-bundle\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.363974 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.363775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4pbl\" (UniqueName: \"kubernetes.io/projected/e1b7bfa2-5769-486b-af68-4bb2868f7264-kube-api-access-x4pbl\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.363974 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.363795 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-federate-client-tls\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.363974 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.363862 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-metrics-client-ca\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.363974 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.363897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-telemeter-client-tls\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465013 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.464983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4pbl\" (UniqueName: \"kubernetes.io/projected/e1b7bfa2-5769-486b-af68-4bb2868f7264-kube-api-access-x4pbl\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465154 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465021 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-federate-client-tls\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465154 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465043 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-metrics-client-ca\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465154 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-telemeter-client-tls\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465154 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465117 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-secret-telemeter-client\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465154 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465148 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465397 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-telemeter-trusted-ca-bundle\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465397 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465316 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-serving-certs-ca-bundle\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.465860 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-metrics-client-ca\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.466018 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.465988 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-serving-certs-ca-bundle\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.466274 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.466247 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1b7bfa2-5769-486b-af68-4bb2868f7264-telemeter-trusted-ca-bundle\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.467586 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.467562 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-secret-telemeter-client\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.467683 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.467667 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-federate-client-tls\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.467758 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.467743 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.467808 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.467790 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1b7bfa2-5769-486b-af68-4bb2868f7264-telemeter-client-tls\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.475428 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.475408 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4pbl\" (UniqueName: \"kubernetes.io/projected/e1b7bfa2-5769-486b-af68-4bb2868f7264-kube-api-access-x4pbl\") pod \"telemeter-client-557ff67dcd-znxkl\" (UID: \"e1b7bfa2-5769-486b-af68-4bb2868f7264\") " pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.581324 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.581255 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" Jul 16 12:46:53.740655 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:53.740565 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-557ff67dcd-znxkl"] Jul 16 12:46:53.744784 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:53.744754 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b7bfa2_5769_486b_af68_4bb2868f7264.slice/crio-995063db39f399c8761b812fe0618c48df92d5cac70854edd6fee76c6678b3ba WatchSource:0}: Error finding container 995063db39f399c8761b812fe0618c48df92d5cac70854edd6fee76c6678b3ba: Status 404 returned error can't find the container with id 995063db39f399c8761b812fe0618c48df92d5cac70854edd6fee76c6678b3ba Jul 16 12:46:54.261577 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.261538 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:46:54.264551 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.264533 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.267742 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.267722 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 12:46:54.270250 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270096 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 12:46:54.270250 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270174 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 12:46:54.270250 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270191 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 12:46:54.270250 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270176 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 12:46:54.270503 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270333 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1qbic4774g87s\"" Jul 16 12:46:54.270503 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270453 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 12:46:54.270503 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270478 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 12:46:54.270656 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270579 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 12:46:54.270656 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270641 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 12:46:54.270784 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.270771 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 12:46:54.271533 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.271516 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 12:46:54.271628 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.271554 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 12:46:54.271689 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.271669 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 12:46:54.271873 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.271852 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pfg7g\"" Jul 16 12:46:54.271956 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.271875 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 12:46:54.273261 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.273243 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 12:46:54.281134 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.281110 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:46:54.354540 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.354508 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d577c4bcb-tkklw"] Jul 16 12:46:54.356697 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.356676 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.371426 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.371397 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d577c4bcb-tkklw"] Jul 16 12:46:54.372872 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.372852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.372973 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.372894 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.372973 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.372940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373067 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.372976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373067 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm79b\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-kube-api-access-jm79b\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373067 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373101 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373319 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373319 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373319 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373319 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373319 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373515 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373345 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373515 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373388 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373515 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373515 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373515 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.373753 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.373533 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474192 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474160 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474192 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474199 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474256 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-oauth-serving-cert\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.474461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474309 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rjzx\" (UniqueName: \"kubernetes.io/projected/f18338bd-774a-4e8d-ba9a-5314ad9505a8-kube-api-access-9rjzx\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.474461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-oauth-config\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.474461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-trusted-ca-bundle\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.474800 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474800 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474800 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm79b\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-kube-api-access-jm79b\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.474800 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474713 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-serving-cert\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.475021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-config\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.475021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.474995 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475021 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475016 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475050 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475108 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475181 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475188 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-service-ca\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475274 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475311 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.475998 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.475922 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.476254 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.476223 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.477859 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.477809 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.478610 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.478585 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.478742 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.478704 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.479054 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.479011 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.479173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.479132 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.480719 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.479977 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.481454 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.481430 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.481585 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.481452 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.481761 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.481735 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.481863 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.481833 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-web-config\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.482179 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.482153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.482293 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.482275 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config-out\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.482649 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.482626 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.485505 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.485483 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm79b\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-kube-api-access-jm79b\") pod \"prometheus-k8s-0\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.486688 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.486661 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" event={"ID":"e1b7bfa2-5769-486b-af68-4bb2868f7264","Type":"ContainerStarted","Data":"995063db39f399c8761b812fe0618c48df92d5cac70854edd6fee76c6678b3ba"} Jul 16 12:46:54.574719 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.574642 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:46:54.575870 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.575790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-service-ca\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.575990 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.575954 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-oauth-serving-cert\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.576051 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.575993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rjzx\" (UniqueName: \"kubernetes.io/projected/f18338bd-774a-4e8d-ba9a-5314ad9505a8-kube-api-access-9rjzx\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.576051 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.576035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-oauth-config\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.576287 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.576057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-trusted-ca-bundle\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.576287 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.576215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-serving-cert\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.576287 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.576271 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-config\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.576731 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.576686 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-service-ca\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.577087 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.577067 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-oauth-serving-cert\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.577146 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.577117 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-trusted-ca-bundle\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.577737 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.577691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-config\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.579244 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.579218 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-serving-cert\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.579654 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.579620 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-oauth-config\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.588666 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.588639 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rjzx\" (UniqueName: \"kubernetes.io/projected/f18338bd-774a-4e8d-ba9a-5314ad9505a8-kube-api-access-9rjzx\") pod \"console-6d577c4bcb-tkklw\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.667259 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.667227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:46:54.731026 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.730703 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:46:54.736854 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:54.736500 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6a5a92_11c4_4161_9581_f5d33fdad16c.slice/crio-e444bf1e963a0a02e37f924017d241a84248fe6c3798a3e1c7616d4049df5650 WatchSource:0}: Error finding container e444bf1e963a0a02e37f924017d241a84248fe6c3798a3e1c7616d4049df5650: Status 404 returned error can't find the container with id e444bf1e963a0a02e37f924017d241a84248fe6c3798a3e1c7616d4049df5650 Jul 16 12:46:54.808638 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:54.808608 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d577c4bcb-tkklw"] Jul 16 12:46:54.812443 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:54.812418 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18338bd_774a_4e8d_ba9a_5314ad9505a8.slice/crio-43b1dc6af8eafc0bdb4b96f109a82244394681083815eb1e9c3f20fdd2284d76 WatchSource:0}: Error finding container 43b1dc6af8eafc0bdb4b96f109a82244394681083815eb1e9c3f20fdd2284d76: Status 404 returned error can't find the container with id 43b1dc6af8eafc0bdb4b96f109a82244394681083815eb1e9c3f20fdd2284d76 Jul 16 12:46:55.493174 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:55.493141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d577c4bcb-tkklw" event={"ID":"f18338bd-774a-4e8d-ba9a-5314ad9505a8","Type":"ContainerStarted","Data":"b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922"} Jul 16 12:46:55.493594 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:55.493197 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d577c4bcb-tkklw" event={"ID":"f18338bd-774a-4e8d-ba9a-5314ad9505a8","Type":"ContainerStarted","Data":"43b1dc6af8eafc0bdb4b96f109a82244394681083815eb1e9c3f20fdd2284d76"} Jul 16 12:46:55.494643 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:55.494617 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerStarted","Data":"e444bf1e963a0a02e37f924017d241a84248fe6c3798a3e1c7616d4049df5650"} Jul 16 12:46:55.515474 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:55.515420 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d577c4bcb-tkklw" podStartSLOduration=1.515402181 podStartE2EDuration="1.515402181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:55.515249115 +0000 UTC m=+101.057855879" watchObservedRunningTime="2026-07-16 12:46:55.515402181 +0000 UTC m=+101.058008930" Jul 16 12:46:56.499336 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:56.499294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" event={"ID":"e1b7bfa2-5769-486b-af68-4bb2868f7264","Type":"ContainerStarted","Data":"b193d3056cde39327c918a181baad2a6a6b213e776d7970c30c917afc8033265"} Jul 16 12:46:57.505290 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:57.505247 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" event={"ID":"e1b7bfa2-5769-486b-af68-4bb2868f7264","Type":"ContainerStarted","Data":"b75e0f28eb42b08663c1bfbfcadfb746fad71360e09d9f4816c619060b2fff7c"} Jul 16 12:46:57.505688 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:57.505316 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" event={"ID":"e1b7bfa2-5769-486b-af68-4bb2868f7264","Type":"ContainerStarted","Data":"d8bea14e5e638c0dbb783b185ff4f2f08f740194548d783ad9fc4303679588a8"} Jul 16 12:46:57.506734 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:57.506707 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" exitCode=0 Jul 16 12:46:57.506837 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:57.506757 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerDied","Data":"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c"} Jul 16 12:46:57.539449 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:57.539402 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-557ff67dcd-znxkl" podStartSLOduration=1.829786098 podStartE2EDuration="4.539382926s" totalImagesPullingTime="2.709596828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:57.538325371 +0000 UTC m=+103.080932193" watchObservedRunningTime="2026-07-16 12:46:57.539382926 +0000 UTC m=+103.081989676" Jul 16 12:46:58.311963 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.311917 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d577c4bcb-tkklw"] Jul 16 12:46:58.346776 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.346746 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76df4bc5-4lcd6"] Jul 16 12:46:58.349130 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.349112 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.372781 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.372759 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76df4bc5-4lcd6"] Jul 16 12:46:58.411417 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.411372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-oauth-config\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.411417 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.411409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-service-ca\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.411633 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.411451 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-oauth-serving-cert\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.411633 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.411517 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-trusted-ca-bundle\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.411633 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.411542 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2vk\" (UniqueName: \"kubernetes.io/projected/93afd687-0e54-4e0c-97cc-5879f7e429a9-kube-api-access-jm2vk\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.411633 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.411568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-config\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.411633 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.411596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-serving-cert\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.513587 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.513542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-trusted-ca-bundle\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.514051 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.513602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2vk\" (UniqueName: \"kubernetes.io/projected/93afd687-0e54-4e0c-97cc-5879f7e429a9-kube-api-access-jm2vk\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.514051 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.513636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-config\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.514051 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.513678 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-serving-cert\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.514051 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.513733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-oauth-config\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.514051 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.513763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-service-ca\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.514051 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.513802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-oauth-serving-cert\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.514623 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.514595 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-oauth-serving-cert\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.515362 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.515337 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-trusted-ca-bundle\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.516363 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.516341 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-config\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.517006 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.516987 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-service-ca\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.518463 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.518440 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-serving-cert\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.518564 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.518503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-oauth-config\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.527226 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.527202 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2vk\" (UniqueName: \"kubernetes.io/projected/93afd687-0e54-4e0c-97cc-5879f7e429a9-kube-api-access-jm2vk\") pod \"console-76df4bc5-4lcd6\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.660361 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.660270 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:46:58.815442 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:58.815416 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76df4bc5-4lcd6"] Jul 16 12:46:58.818555 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:46:58.818527 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93afd687_0e54_4e0c_97cc_5879f7e429a9.slice/crio-e519d6165a8398d8ffc486c553b0e7dbd6f5ac44524088a289fc096f8c58e62a WatchSource:0}: Error finding container e519d6165a8398d8ffc486c553b0e7dbd6f5ac44524088a289fc096f8c58e62a: Status 404 returned error can't find the container with id e519d6165a8398d8ffc486c553b0e7dbd6f5ac44524088a289fc096f8c58e62a Jul 16 12:46:59.518263 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:59.518216 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76df4bc5-4lcd6" event={"ID":"93afd687-0e54-4e0c-97cc-5879f7e429a9","Type":"ContainerStarted","Data":"79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8"} Jul 16 12:46:59.518711 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:59.518287 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76df4bc5-4lcd6" event={"ID":"93afd687-0e54-4e0c-97cc-5879f7e429a9","Type":"ContainerStarted","Data":"e519d6165a8398d8ffc486c553b0e7dbd6f5ac44524088a289fc096f8c58e62a"} Jul 16 12:46:59.555914 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:46:59.555861 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76df4bc5-4lcd6" podStartSLOduration=1.555840952 podStartE2EDuration="1.555840952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:46:59.554314881 +0000 UTC m=+105.096921634" watchObservedRunningTime="2026-07-16 12:46:59.555840952 +0000 UTC m=+105.098447699" Jul 16 12:47:01.531673 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:01.531629 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerStarted","Data":"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8"} Jul 16 12:47:01.532146 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:01.531724 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerStarted","Data":"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283"} Jul 16 12:47:04.546357 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:04.546317 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerStarted","Data":"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e"} Jul 16 12:47:04.546850 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:04.546397 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerStarted","Data":"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484"} Jul 16 12:47:04.546850 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:04.546443 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerStarted","Data":"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485"} Jul 16 12:47:04.546850 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:04.546484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerStarted","Data":"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca"} Jul 16 12:47:04.575194 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:04.575163 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:47:04.587044 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:04.587000 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.954766362 podStartE2EDuration="10.586983167s" totalImagesPullingTime="7.610150052s" totalInitContainerRuntime="22.066753ms" isStatefulPod=false podCreationTimestamp="2026-07-16 12:46:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:47:04.585708401 +0000 UTC m=+110.128315147" watchObservedRunningTime="2026-07-16 12:47:04.586983167 +0000 UTC m=+110.129589914" Jul 16 12:47:04.667731 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:04.667695 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:47:08.661331 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:08.661293 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:47:08.661727 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:08.661357 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:47:08.665621 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:08.665601 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:47:09.569527 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:09.569500 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:47:09.627766 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:09.627734 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7454ffd75d-kwpms"] Jul 16 12:47:22.622461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:22.622428 2576 generic.go:356] "Generic (PLEG): container finished" podID="192c0582-645d-451e-8f1a-edef1f89bbbd" containerID="8a7f81c9bb44bc7dff752936627ad558c779aaaf40502e56fd07cce4abcd2ff8" exitCode=0 Jul 16 12:47:22.622808 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:22.622485 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" event={"ID":"192c0582-645d-451e-8f1a-edef1f89bbbd","Type":"ContainerDied","Data":"8a7f81c9bb44bc7dff752936627ad558c779aaaf40502e56fd07cce4abcd2ff8"} Jul 16 12:47:22.622808 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:22.622781 2576 scope.go:117] "RemoveContainer" containerID="8a7f81c9bb44bc7dff752936627ad558c779aaaf40502e56fd07cce4abcd2ff8" Jul 16 12:47:23.332222 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.332178 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d577c4bcb-tkklw" podUID="f18338bd-774a-4e8d-ba9a-5314ad9505a8" containerName="console" containerID="cri-o://b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922" gracePeriod=15 Jul 16 12:47:23.591163 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.591112 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:47:23.635576 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.635539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-6cmbz" event={"ID":"192c0582-645d-451e-8f1a-edef1f89bbbd","Type":"ContainerStarted","Data":"c073fbf67d1597a2c88c02e2cb55a421a4ada0d8186a2b355eb237d85193206d"} Jul 16 12:47:23.637058 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.637026 2576 generic.go:356] "Generic (PLEG): container finished" podID="f18338bd-774a-4e8d-ba9a-5314ad9505a8" containerID="b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922" exitCode=0 Jul 16 12:47:23.637173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.637074 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d577c4bcb-tkklw" Jul 16 12:47:23.637173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.637100 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d577c4bcb-tkklw" event={"ID":"f18338bd-774a-4e8d-ba9a-5314ad9505a8","Type":"ContainerDied","Data":"b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922"} Jul 16 12:47:23.637173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.637141 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d577c4bcb-tkklw" event={"ID":"f18338bd-774a-4e8d-ba9a-5314ad9505a8","Type":"ContainerDied","Data":"43b1dc6af8eafc0bdb4b96f109a82244394681083815eb1e9c3f20fdd2284d76"} Jul 16 12:47:23.637173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.637167 2576 scope.go:117] "RemoveContainer" containerID="b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922" Jul 16 12:47:23.646315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.646296 2576 scope.go:117] "RemoveContainer" containerID="b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922" Jul 16 12:47:23.646571 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:47:23.646555 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922\": container with ID starting with b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922 not found: ID does not exist" containerID="b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922" Jul 16 12:47:23.646627 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.646582 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922"} err="failed to get container status \"b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922\": rpc error: code = NotFound desc = could not find container \"b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922\": container with ID starting with b070dc69973b7d5f400bfdae892b2fd42faed0792c09ba0bf8939ba2a5eec922 not found: ID does not exist" Jul 16 12:47:23.751417 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.751387 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-serving-cert\") pod \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " Jul 16 12:47:23.751564 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.751443 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rjzx\" (UniqueName: \"kubernetes.io/projected/f18338bd-774a-4e8d-ba9a-5314ad9505a8-kube-api-access-9rjzx\") pod \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " Jul 16 12:47:23.751564 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.751468 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-config\") pod \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " Jul 16 12:47:23.751564 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.751495 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-trusted-ca-bundle\") pod \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " Jul 16 12:47:23.751564 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.751519 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-oauth-config\") pod \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " Jul 16 12:47:23.751564 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.751549 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-oauth-serving-cert\") pod \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " Jul 16 12:47:23.751802 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.751588 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-service-ca\") pod \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\" (UID: \"f18338bd-774a-4e8d-ba9a-5314ad9505a8\") " Jul 16 12:47:23.752130 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.752096 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-service-ca" (OuterVolumeSpecName: "service-ca") pod "f18338bd-774a-4e8d-ba9a-5314ad9505a8" (UID: "f18338bd-774a-4e8d-ba9a-5314ad9505a8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:23.752236 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.752106 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-config" (OuterVolumeSpecName: "console-config") pod "f18338bd-774a-4e8d-ba9a-5314ad9505a8" (UID: "f18338bd-774a-4e8d-ba9a-5314ad9505a8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:23.752236 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.752131 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f18338bd-774a-4e8d-ba9a-5314ad9505a8" (UID: "f18338bd-774a-4e8d-ba9a-5314ad9505a8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:23.752318 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.752229 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f18338bd-774a-4e8d-ba9a-5314ad9505a8" (UID: "f18338bd-774a-4e8d-ba9a-5314ad9505a8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:23.753640 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.753620 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f18338bd-774a-4e8d-ba9a-5314ad9505a8" (UID: "f18338bd-774a-4e8d-ba9a-5314ad9505a8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:23.753740 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.753671 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f18338bd-774a-4e8d-ba9a-5314ad9505a8-kube-api-access-9rjzx" (OuterVolumeSpecName: "kube-api-access-9rjzx") pod "f18338bd-774a-4e8d-ba9a-5314ad9505a8" (UID: "f18338bd-774a-4e8d-ba9a-5314ad9505a8"). InnerVolumeSpecName "kube-api-access-9rjzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:23.753797 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.753743 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f18338bd-774a-4e8d-ba9a-5314ad9505a8" (UID: "f18338bd-774a-4e8d-ba9a-5314ad9505a8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:23.852511 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.852443 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rjzx\" (UniqueName: \"kubernetes.io/projected/f18338bd-774a-4e8d-ba9a-5314ad9505a8-kube-api-access-9rjzx\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:23.852511 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.852475 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-config\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:23.852511 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.852485 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-trusted-ca-bundle\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:23.852511 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.852495 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-oauth-config\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:23.852511 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.852504 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-oauth-serving-cert\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:23.852511 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.852512 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f18338bd-774a-4e8d-ba9a-5314ad9505a8-service-ca\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:23.852511 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.852520 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f18338bd-774a-4e8d-ba9a-5314ad9505a8-console-serving-cert\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:23.960688 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.960663 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d577c4bcb-tkklw"] Jul 16 12:47:23.965614 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:23.965593 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d577c4bcb-tkklw"] Jul 16 12:47:24.985745 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:24.985717 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f18338bd-774a-4e8d-ba9a-5314ad9505a8" path="/var/lib/kubelet/pods/f18338bd-774a-4e8d-ba9a-5314ad9505a8/volumes" Jul 16 12:47:34.655407 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.655365 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7454ffd75d-kwpms" podUID="5c621b5c-23ac-41a6-b895-b7974b08d8f4" containerName="console" containerID="cri-o://ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb" gracePeriod=15 Jul 16 12:47:34.896256 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.896228 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:47:34.934006 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.933977 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-trusted-ca-bundle\") pod \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " Jul 16 12:47:34.934267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934242 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxm64\" (UniqueName: \"kubernetes.io/projected/5c621b5c-23ac-41a6-b895-b7974b08d8f4-kube-api-access-cxm64\") pod \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " Jul 16 12:47:34.934393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934314 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-oauth-config\") pod \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " Jul 16 12:47:34.934393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934346 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-config\") pod \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " Jul 16 12:47:34.934393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934359 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5c621b5c-23ac-41a6-b895-b7974b08d8f4" (UID: "5c621b5c-23ac-41a6-b895-b7974b08d8f4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:34.934393 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934376 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-service-ca\") pod \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " Jul 16 12:47:34.934602 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934421 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-oauth-serving-cert\") pod \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " Jul 16 12:47:34.934602 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934466 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-serving-cert\") pod \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\" (UID: \"5c621b5c-23ac-41a6-b895-b7974b08d8f4\") " Jul 16 12:47:34.934737 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934712 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-config" (OuterVolumeSpecName: "console-config") pod "5c621b5c-23ac-41a6-b895-b7974b08d8f4" (UID: "5c621b5c-23ac-41a6-b895-b7974b08d8f4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:34.934737 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934727 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-service-ca" (OuterVolumeSpecName: "service-ca") pod "5c621b5c-23ac-41a6-b895-b7974b08d8f4" (UID: "5c621b5c-23ac-41a6-b895-b7974b08d8f4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:34.934980 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934961 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-config\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:34.935053 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.934985 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-service-ca\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:34.935053 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.935000 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-trusted-ca-bundle\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:34.935053 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.935011 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5c621b5c-23ac-41a6-b895-b7974b08d8f4" (UID: "5c621b5c-23ac-41a6-b895-b7974b08d8f4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:47:34.936789 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.936766 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c621b5c-23ac-41a6-b895-b7974b08d8f4-kube-api-access-cxm64" (OuterVolumeSpecName: "kube-api-access-cxm64") pod "5c621b5c-23ac-41a6-b895-b7974b08d8f4" (UID: "5c621b5c-23ac-41a6-b895-b7974b08d8f4"). InnerVolumeSpecName "kube-api-access-cxm64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:47:34.936895 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.936834 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5c621b5c-23ac-41a6-b895-b7974b08d8f4" (UID: "5c621b5c-23ac-41a6-b895-b7974b08d8f4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:34.936963 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:34.936905 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5c621b5c-23ac-41a6-b895-b7974b08d8f4" (UID: "5c621b5c-23ac-41a6-b895-b7974b08d8f4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:47:35.035612 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.035584 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-oauth-config\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:35.035612 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.035607 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5c621b5c-23ac-41a6-b895-b7974b08d8f4-oauth-serving-cert\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:35.035746 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.035618 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5c621b5c-23ac-41a6-b895-b7974b08d8f4-console-serving-cert\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:35.035746 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.035626 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxm64\" (UniqueName: \"kubernetes.io/projected/5c621b5c-23ac-41a6-b895-b7974b08d8f4-kube-api-access-cxm64\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:47:35.686763 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.686723 2576 generic.go:356] "Generic (PLEG): container finished" podID="5c621b5c-23ac-41a6-b895-b7974b08d8f4" containerID="ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb" exitCode=0 Jul 16 12:47:35.687161 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.686790 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7454ffd75d-kwpms" Jul 16 12:47:35.687161 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.686826 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7454ffd75d-kwpms" event={"ID":"5c621b5c-23ac-41a6-b895-b7974b08d8f4","Type":"ContainerDied","Data":"ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb"} Jul 16 12:47:35.687161 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.686883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7454ffd75d-kwpms" event={"ID":"5c621b5c-23ac-41a6-b895-b7974b08d8f4","Type":"ContainerDied","Data":"bb13cb9059109036e3ec8b8668d74e5aa8e6d66a3280bbd062f53bcddc33f0f7"} Jul 16 12:47:35.687161 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.686903 2576 scope.go:117] "RemoveContainer" containerID="ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb" Jul 16 12:47:35.694676 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.694656 2576 scope.go:117] "RemoveContainer" containerID="ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb" Jul 16 12:47:35.694944 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:47:35.694924 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb\": container with ID starting with ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb not found: ID does not exist" containerID="ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb" Jul 16 12:47:35.695010 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.694951 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb"} err="failed to get container status \"ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb\": rpc error: code = NotFound desc = could not find container \"ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb\": container with ID starting with ce2010f24dc16134738817459b766cdf895b25111cfffb96ac22530c3abbe0fb not found: ID does not exist" Jul 16 12:47:35.720998 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.720969 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7454ffd75d-kwpms"] Jul 16 12:47:35.725296 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:35.725273 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7454ffd75d-kwpms"] Jul 16 12:47:36.985387 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:36.985341 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c621b5c-23ac-41a6-b895-b7974b08d8f4" path="/var/lib/kubelet/pods/5c621b5c-23ac-41a6-b895-b7974b08d8f4/volumes" Jul 16 12:47:54.575803 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:54.575769 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:47:54.594419 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:54.594397 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:47:54.774879 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:47:54.774853 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:12.927230 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:12.927157 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:48:12.927773 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:12.927451 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="prometheus" containerID="cri-o://04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" gracePeriod=600 Jul 16 12:48:12.927773 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:12.927482 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy" containerID="cri-o://4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" gracePeriod=600 Jul 16 12:48:12.927773 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:12.927500 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy-web" containerID="cri-o://40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" gracePeriod=600 Jul 16 12:48:12.927773 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:12.927498 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="thanos-sidecar" containerID="cri-o://c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" gracePeriod=600 Jul 16 12:48:12.927773 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:12.927540 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="config-reloader" containerID="cri-o://5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" gracePeriod=600 Jul 16 12:48:12.927773 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:12.927613 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" gracePeriod=600 Jul 16 12:48:13.167451 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.167427 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:13.350199 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350121 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm79b\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-kube-api-access-jm79b\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350199 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350168 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-metrics-client-ca\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350199 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350185 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-1\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350425 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350205 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-2\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350425 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350306 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-0\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350425 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350356 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-tls\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350425 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350384 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-db\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350425 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350423 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-trusted-ca-bundle\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350676 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350449 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-kube-rbac-proxy\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350676 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350560 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:13.350676 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350561 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:13.350676 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350619 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-kubelet-serving-ca-bundle\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350917 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350693 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350917 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350755 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config-out\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350917 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350791 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-serving-certs-ca-bundle\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350917 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350801 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:13.350917 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350842 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-metrics-client-certs\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350917 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350867 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-grpc-tls\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.350917 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350890 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-web-config\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.351248 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350926 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.351248 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350953 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-tls-assets\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.351248 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350977 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.351248 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.351031 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-thanos-prometheus-http-client-file\") pod \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\" (UID: \"0f6a5a92-11c4-4161-9581-f5d33fdad16c\") " Jul 16 12:48:13.351456 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.351285 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-metrics-client-ca\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.351456 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.351301 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.351456 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.351315 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.352188 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.350975 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:13.352188 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.351331 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:13.352380 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.351691 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:13.352380 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.351700 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:13.352380 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.352052 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:13.353002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.352964 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-kube-api-access-jm79b" (OuterVolumeSpecName: "kube-api-access-jm79b") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "kube-api-access-jm79b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:48:13.353198 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.353170 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.354429 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.354392 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.354531 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.354453 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config" (OuterVolumeSpecName: "config") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.354531 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.354518 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config-out" (OuterVolumeSpecName: "config-out") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:48:13.354626 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.354526 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.354736 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.354719 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.355211 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.355184 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.355652 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.355633 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:48:13.355905 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.355884 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.356069 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.356048 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.364180 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.364162 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-web-config" (OuterVolumeSpecName: "web-config") pod "0f6a5a92-11c4-4161-9581-f5d33fdad16c" (UID: "0f6a5a92-11c4-4161-9581-f5d33fdad16c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:13.452607 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452578 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452607 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452605 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm79b\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-kube-api-access-jm79b\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452620 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452633 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452644 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-k8s-db\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452657 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452670 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-kube-rbac-proxy\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452682 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452695 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452707 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f6a5a92-11c4-4161-9581-f5d33fdad16c-config-out\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452719 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0f6a5a92-11c4-4161-9581-f5d33fdad16c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452731 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-metrics-client-certs\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452743 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-grpc-tls\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452756 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-web-config\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452769 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452783 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f6a5a92-11c4-4161-9581-f5d33fdad16c-tls-assets\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.452793 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.452796 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0f6a5a92-11c4-4161-9581-f5d33fdad16c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:13.837519 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837488 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" exitCode=0 Jul 16 12:48:13.837519 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837514 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" exitCode=0 Jul 16 12:48:13.837519 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837521 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" exitCode=0 Jul 16 12:48:13.837519 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837528 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" exitCode=0 Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837534 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" exitCode=0 Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837539 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" exitCode=0 Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837569 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerDied","Data":"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e"} Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837684 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerDied","Data":"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484"} Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerDied","Data":"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485"} Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837746 2576 scope.go:117] "RemoveContainer" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837774 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerDied","Data":"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca"} Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837835 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerDied","Data":"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8"} Jul 16 12:48:13.837886 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837879 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerDied","Data":"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283"} Jul 16 12:48:13.838303 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.837918 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0f6a5a92-11c4-4161-9581-f5d33fdad16c","Type":"ContainerDied","Data":"e444bf1e963a0a02e37f924017d241a84248fe6c3798a3e1c7616d4049df5650"} Jul 16 12:48:13.845741 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.845721 2576 scope.go:117] "RemoveContainer" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" Jul 16 12:48:13.852408 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.852391 2576 scope.go:117] "RemoveContainer" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" Jul 16 12:48:13.858993 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.858976 2576 scope.go:117] "RemoveContainer" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" Jul 16 12:48:13.865176 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.865162 2576 scope.go:117] "RemoveContainer" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" Jul 16 12:48:13.872332 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.872304 2576 scope.go:117] "RemoveContainer" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" Jul 16 12:48:13.873301 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.873284 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:48:13.878008 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.877987 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:48:13.879659 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.879645 2576 scope.go:117] "RemoveContainer" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" Jul 16 12:48:13.885912 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.885897 2576 scope.go:117] "RemoveContainer" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" Jul 16 12:48:13.886139 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:48:13.886121 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": container with ID starting with ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e not found: ID does not exist" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" Jul 16 12:48:13.886193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.886145 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e"} err="failed to get container status \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": rpc error: code = NotFound desc = could not find container \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": container with ID starting with ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e not found: ID does not exist" Jul 16 12:48:13.886193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.886163 2576 scope.go:117] "RemoveContainer" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" Jul 16 12:48:13.886393 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:48:13.886376 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": container with ID starting with 4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484 not found: ID does not exist" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" Jul 16 12:48:13.886432 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.886398 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484"} err="failed to get container status \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": rpc error: code = NotFound desc = could not find container \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": container with ID starting with 4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484 not found: ID does not exist" Jul 16 12:48:13.886432 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.886412 2576 scope.go:117] "RemoveContainer" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" Jul 16 12:48:13.886641 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:48:13.886626 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": container with ID starting with 40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485 not found: ID does not exist" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" Jul 16 12:48:13.886681 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.886645 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485"} err="failed to get container status \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": rpc error: code = NotFound desc = could not find container \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": container with ID starting with 40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485 not found: ID does not exist" Jul 16 12:48:13.886681 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.886659 2576 scope.go:117] "RemoveContainer" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" Jul 16 12:48:13.886857 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:48:13.886842 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": container with ID starting with c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca not found: ID does not exist" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" Jul 16 12:48:13.886899 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.886860 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca"} err="failed to get container status \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": rpc error: code = NotFound desc = could not find container \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": container with ID starting with c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca not found: ID does not exist" Jul 16 12:48:13.886899 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.886872 2576 scope.go:117] "RemoveContainer" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" Jul 16 12:48:13.887101 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:48:13.887083 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": container with ID starting with 5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8 not found: ID does not exist" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" Jul 16 12:48:13.887150 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887106 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8"} err="failed to get container status \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": rpc error: code = NotFound desc = could not find container \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": container with ID starting with 5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8 not found: ID does not exist" Jul 16 12:48:13.887150 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887122 2576 scope.go:117] "RemoveContainer" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" Jul 16 12:48:13.887359 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:48:13.887338 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": container with ID starting with 04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283 not found: ID does not exist" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" Jul 16 12:48:13.887461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887361 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283"} err="failed to get container status \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": rpc error: code = NotFound desc = could not find container \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": container with ID starting with 04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283 not found: ID does not exist" Jul 16 12:48:13.887461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887373 2576 scope.go:117] "RemoveContainer" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" Jul 16 12:48:13.887568 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:48:13.887554 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": container with ID starting with 2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c not found: ID does not exist" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" Jul 16 12:48:13.887604 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887571 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c"} err="failed to get container status \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": rpc error: code = NotFound desc = could not find container \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": container with ID starting with 2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c not found: ID does not exist" Jul 16 12:48:13.887604 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887585 2576 scope.go:117] "RemoveContainer" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" Jul 16 12:48:13.887734 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887718 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e"} err="failed to get container status \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": rpc error: code = NotFound desc = could not find container \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": container with ID starting with ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e not found: ID does not exist" Jul 16 12:48:13.887777 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887734 2576 scope.go:117] "RemoveContainer" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" Jul 16 12:48:13.887957 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887939 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484"} err="failed to get container status \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": rpc error: code = NotFound desc = could not find container \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": container with ID starting with 4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484 not found: ID does not exist" Jul 16 12:48:13.888009 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.887958 2576 scope.go:117] "RemoveContainer" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" Jul 16 12:48:13.888127 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888112 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485"} err="failed to get container status \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": rpc error: code = NotFound desc = could not find container \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": container with ID starting with 40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485 not found: ID does not exist" Jul 16 12:48:13.888168 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888127 2576 scope.go:117] "RemoveContainer" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" Jul 16 12:48:13.888358 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888341 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca"} err="failed to get container status \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": rpc error: code = NotFound desc = could not find container \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": container with ID starting with c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca not found: ID does not exist" Jul 16 12:48:13.888408 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888359 2576 scope.go:117] "RemoveContainer" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" Jul 16 12:48:13.888571 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888552 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8"} err="failed to get container status \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": rpc error: code = NotFound desc = could not find container \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": container with ID starting with 5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8 not found: ID does not exist" Jul 16 12:48:13.888608 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888572 2576 scope.go:117] "RemoveContainer" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" Jul 16 12:48:13.888745 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888732 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283"} err="failed to get container status \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": rpc error: code = NotFound desc = could not find container \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": container with ID starting with 04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283 not found: ID does not exist" Jul 16 12:48:13.888791 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888745 2576 scope.go:117] "RemoveContainer" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" Jul 16 12:48:13.888950 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888932 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c"} err="failed to get container status \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": rpc error: code = NotFound desc = could not find container \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": container with ID starting with 2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c not found: ID does not exist" Jul 16 12:48:13.889012 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.888953 2576 scope.go:117] "RemoveContainer" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" Jul 16 12:48:13.889119 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889104 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e"} err="failed to get container status \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": rpc error: code = NotFound desc = could not find container \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": container with ID starting with ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e not found: ID does not exist" Jul 16 12:48:13.889156 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889121 2576 scope.go:117] "RemoveContainer" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" Jul 16 12:48:13.889326 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889310 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484"} err="failed to get container status \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": rpc error: code = NotFound desc = could not find container \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": container with ID starting with 4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484 not found: ID does not exist" Jul 16 12:48:13.889375 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889328 2576 scope.go:117] "RemoveContainer" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" Jul 16 12:48:13.889518 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889496 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485"} err="failed to get container status \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": rpc error: code = NotFound desc = could not find container \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": container with ID starting with 40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485 not found: ID does not exist" Jul 16 12:48:13.889518 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889516 2576 scope.go:117] "RemoveContainer" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" Jul 16 12:48:13.889707 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889689 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca"} err="failed to get container status \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": rpc error: code = NotFound desc = could not find container \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": container with ID starting with c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca not found: ID does not exist" Jul 16 12:48:13.889752 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889709 2576 scope.go:117] "RemoveContainer" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" Jul 16 12:48:13.889954 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889938 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8"} err="failed to get container status \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": rpc error: code = NotFound desc = could not find container \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": container with ID starting with 5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8 not found: ID does not exist" Jul 16 12:48:13.889954 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.889953 2576 scope.go:117] "RemoveContainer" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" Jul 16 12:48:13.890198 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.890183 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283"} err="failed to get container status \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": rpc error: code = NotFound desc = could not find container \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": container with ID starting with 04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283 not found: ID does not exist" Jul 16 12:48:13.890261 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.890199 2576 scope.go:117] "RemoveContainer" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" Jul 16 12:48:13.890394 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.890375 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c"} err="failed to get container status \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": rpc error: code = NotFound desc = could not find container \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": container with ID starting with 2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c not found: ID does not exist" Jul 16 12:48:13.890394 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.890393 2576 scope.go:117] "RemoveContainer" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" Jul 16 12:48:13.890629 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.890613 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e"} err="failed to get container status \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": rpc error: code = NotFound desc = could not find container \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": container with ID starting with ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e not found: ID does not exist" Jul 16 12:48:13.890629 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.890629 2576 scope.go:117] "RemoveContainer" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" Jul 16 12:48:13.890801 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.890783 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484"} err="failed to get container status \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": rpc error: code = NotFound desc = could not find container \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": container with ID starting with 4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484 not found: ID does not exist" Jul 16 12:48:13.890891 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.890803 2576 scope.go:117] "RemoveContainer" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" Jul 16 12:48:13.891053 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891037 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485"} err="failed to get container status \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": rpc error: code = NotFound desc = could not find container \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": container with ID starting with 40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485 not found: ID does not exist" Jul 16 12:48:13.891119 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891054 2576 scope.go:117] "RemoveContainer" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" Jul 16 12:48:13.891270 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891251 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca"} err="failed to get container status \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": rpc error: code = NotFound desc = could not find container \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": container with ID starting with c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca not found: ID does not exist" Jul 16 12:48:13.891315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891271 2576 scope.go:117] "RemoveContainer" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" Jul 16 12:48:13.891473 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891455 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8"} err="failed to get container status \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": rpc error: code = NotFound desc = could not find container \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": container with ID starting with 5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8 not found: ID does not exist" Jul 16 12:48:13.891512 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891474 2576 scope.go:117] "RemoveContainer" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" Jul 16 12:48:13.891647 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891621 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283"} err="failed to get container status \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": rpc error: code = NotFound desc = could not find container \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": container with ID starting with 04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283 not found: ID does not exist" Jul 16 12:48:13.891647 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891635 2576 scope.go:117] "RemoveContainer" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" Jul 16 12:48:13.891870 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891851 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c"} err="failed to get container status \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": rpc error: code = NotFound desc = could not find container \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": container with ID starting with 2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c not found: ID does not exist" Jul 16 12:48:13.891914 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.891872 2576 scope.go:117] "RemoveContainer" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" Jul 16 12:48:13.892071 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892057 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e"} err="failed to get container status \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": rpc error: code = NotFound desc = could not find container \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": container with ID starting with ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e not found: ID does not exist" Jul 16 12:48:13.892107 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892071 2576 scope.go:117] "RemoveContainer" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" Jul 16 12:48:13.892280 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892264 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484"} err="failed to get container status \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": rpc error: code = NotFound desc = could not find container \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": container with ID starting with 4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484 not found: ID does not exist" Jul 16 12:48:13.892318 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892281 2576 scope.go:117] "RemoveContainer" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" Jul 16 12:48:13.892478 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892460 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485"} err="failed to get container status \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": rpc error: code = NotFound desc = could not find container \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": container with ID starting with 40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485 not found: ID does not exist" Jul 16 12:48:13.892516 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892480 2576 scope.go:117] "RemoveContainer" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" Jul 16 12:48:13.892672 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892656 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca"} err="failed to get container status \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": rpc error: code = NotFound desc = could not find container \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": container with ID starting with c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca not found: ID does not exist" Jul 16 12:48:13.892672 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892671 2576 scope.go:117] "RemoveContainer" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" Jul 16 12:48:13.892865 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892849 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8"} err="failed to get container status \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": rpc error: code = NotFound desc = could not find container \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": container with ID starting with 5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8 not found: ID does not exist" Jul 16 12:48:13.892918 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.892865 2576 scope.go:117] "RemoveContainer" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" Jul 16 12:48:13.893064 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893048 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283"} err="failed to get container status \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": rpc error: code = NotFound desc = could not find container \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": container with ID starting with 04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283 not found: ID does not exist" Jul 16 12:48:13.893104 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893063 2576 scope.go:117] "RemoveContainer" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" Jul 16 12:48:13.893275 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893258 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c"} err="failed to get container status \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": rpc error: code = NotFound desc = could not find container \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": container with ID starting with 2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c not found: ID does not exist" Jul 16 12:48:13.893322 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893275 2576 scope.go:117] "RemoveContainer" containerID="ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e" Jul 16 12:48:13.893470 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893450 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e"} err="failed to get container status \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": rpc error: code = NotFound desc = could not find container \"ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e\": container with ID starting with ebfedecd26e351ff11db236de3d2c10edab7ae0e0bcf9906e6318e3a3c9f649e not found: ID does not exist" Jul 16 12:48:13.893508 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893470 2576 scope.go:117] "RemoveContainer" containerID="4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484" Jul 16 12:48:13.893686 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893670 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484"} err="failed to get container status \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": rpc error: code = NotFound desc = could not find container \"4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484\": container with ID starting with 4e6a1e4aefbba73c57e1e65ed7cf2d16fcb894df69e7b8dbd66952f83a2a6484 not found: ID does not exist" Jul 16 12:48:13.893729 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893686 2576 scope.go:117] "RemoveContainer" containerID="40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485" Jul 16 12:48:13.893924 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893906 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485"} err="failed to get container status \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": rpc error: code = NotFound desc = could not find container \"40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485\": container with ID starting with 40fdb735c5aa603ac98da7ce569e12a4439cbf72d4772a3195228b8544c9f485 not found: ID does not exist" Jul 16 12:48:13.894002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.893925 2576 scope.go:117] "RemoveContainer" containerID="c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca" Jul 16 12:48:13.894134 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.894117 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca"} err="failed to get container status \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": rpc error: code = NotFound desc = could not find container \"c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca\": container with ID starting with c152f7c190e56383ae2e14017e18f4eac7e5d32abc32b3dd6847c18a78a001ca not found: ID does not exist" Jul 16 12:48:13.894178 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.894134 2576 scope.go:117] "RemoveContainer" containerID="5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8" Jul 16 12:48:13.894339 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.894321 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8"} err="failed to get container status \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": rpc error: code = NotFound desc = could not find container \"5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8\": container with ID starting with 5cc6179d76c184e89b48683236a674ac8e7c054a6fd0ba28d3f9ab9fb2ebe9e8 not found: ID does not exist" Jul 16 12:48:13.894379 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.894339 2576 scope.go:117] "RemoveContainer" containerID="04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283" Jul 16 12:48:13.894543 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.894527 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283"} err="failed to get container status \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": rpc error: code = NotFound desc = could not find container \"04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283\": container with ID starting with 04737b5558475c792da9e46a70635ff7780266f181200e395fa26bcd7841d283 not found: ID does not exist" Jul 16 12:48:13.894581 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.894544 2576 scope.go:117] "RemoveContainer" containerID="2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c" Jul 16 12:48:13.894735 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.894716 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c"} err="failed to get container status \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": rpc error: code = NotFound desc = could not find container \"2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c\": container with ID starting with 2634d3531379579ff5f876c5caaf27a68b1b5bfddc8a5428bd39491dfcca772c not found: ID does not exist" Jul 16 12:48:13.925566 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925537 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:48:13.925810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925798 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="config-reloader" Jul 16 12:48:13.925810 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925812 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="config-reloader" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925841 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f18338bd-774a-4e8d-ba9a-5314ad9505a8" containerName="console" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925850 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f18338bd-774a-4e8d-ba9a-5314ad9505a8" containerName="console" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925864 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy-thanos" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925870 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy-thanos" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925879 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="init-config-reloader" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925885 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="init-config-reloader" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925894 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c621b5c-23ac-41a6-b895-b7974b08d8f4" containerName="console" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925899 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c621b5c-23ac-41a6-b895-b7974b08d8f4" containerName="console" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925905 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925910 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925916 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="prometheus" Jul 16 12:48:13.925920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925921 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="prometheus" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925927 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="thanos-sidecar" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925932 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="thanos-sidecar" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925940 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy-web" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.925946 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy-web" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.926003 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy-thanos" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.926012 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy-web" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.926017 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c621b5c-23ac-41a6-b895-b7974b08d8f4" containerName="console" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.926022 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="prometheus" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.926028 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="thanos-sidecar" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.926034 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="kube-rbac-proxy" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.926039 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" containerName="config-reloader" Jul 16 12:48:13.926315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.926046 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f18338bd-774a-4e8d-ba9a-5314ad9505a8" containerName="console" Jul 16 12:48:13.931242 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.931227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:13.935450 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.935428 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 16 12:48:13.935536 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.935475 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 16 12:48:13.935536 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.935478 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 16 12:48:13.935982 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.935966 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-pfg7g\"" Jul 16 12:48:13.936738 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.936595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 16 12:48:13.936862 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.936812 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 16 12:48:13.938158 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.938131 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 16 12:48:13.938336 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.938320 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1qbic4774g87s\"" Jul 16 12:48:13.938441 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.938355 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 16 12:48:13.938572 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.938542 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 16 12:48:13.938804 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.938642 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 16 12:48:13.938804 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.938703 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 16 12:48:13.938804 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.938766 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 16 12:48:13.938804 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.938784 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 16 12:48:13.961023 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.961002 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 16 12:48:13.961252 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.961238 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 16 12:48:13.961388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.961358 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 16 12:48:13.962173 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:13.962155 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:48:14.057867 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.057845 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058003 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.057874 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058003 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.057892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058003 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.057909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058003 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.057976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-config\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058003 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058160 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b12af3b5-3373-447b-9103-15a422c067e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058160 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058160 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058069 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058160 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058092 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058160 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058111 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058160 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvgvj\" (UniqueName: \"kubernetes.io/projected/b12af3b5-3373-447b-9103-15a422c067e9-kube-api-access-zvgvj\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058214 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b12af3b5-3373-447b-9103-15a422c067e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058356 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058388 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058387 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.058597 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.058409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159506 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159506 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159466 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159506 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159486 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159552 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-config\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159581 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b12af3b5-3373-447b-9103-15a422c067e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159633 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159677 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.159743 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159722 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159763 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvgvj\" (UniqueName: \"kubernetes.io/projected/b12af3b5-3373-447b-9103-15a422c067e9-kube-api-access-zvgvj\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159793 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159924 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b12af3b5-3373-447b-9103-15a422c067e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.159990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.160014 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160151 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.160046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.160949 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.160739 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.161268 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.161245 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.161907 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.161888 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.162873 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.162493 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.162873 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.162673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.162873 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.162800 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.163376 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.163352 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.164002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.163648 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.164002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.163687 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.164002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.163859 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.164002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.163876 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-config\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.164002 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.163953 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-web-config\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.164289 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.164040 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b12af3b5-3373-447b-9103-15a422c067e9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.164345 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.164302 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.164968 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.164944 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.165508 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.165490 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.165662 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.165642 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b12af3b5-3373-447b-9103-15a422c067e9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.165904 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.165887 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b12af3b5-3373-447b-9103-15a422c067e9-config-out\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.166028 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.166012 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b12af3b5-3373-447b-9103-15a422c067e9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.173594 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.173573 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvgvj\" (UniqueName: \"kubernetes.io/projected/b12af3b5-3373-447b-9103-15a422c067e9-kube-api-access-zvgvj\") pod \"prometheus-k8s-0\" (UID: \"b12af3b5-3373-447b-9103-15a422c067e9\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.254567 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.254546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:14.395337 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.395315 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 16 12:48:14.397095 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:48:14.397071 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12af3b5_3373_447b_9103_15a422c067e9.slice/crio-8cd154f11111b3e6b5f52f5d24bd19518b7c328bda79ef8dd09e7878f8a4ce00 WatchSource:0}: Error finding container 8cd154f11111b3e6b5f52f5d24bd19518b7c328bda79ef8dd09e7878f8a4ce00: Status 404 returned error can't find the container with id 8cd154f11111b3e6b5f52f5d24bd19518b7c328bda79ef8dd09e7878f8a4ce00 Jul 16 12:48:14.843503 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.843470 2576 generic.go:356] "Generic (PLEG): container finished" podID="b12af3b5-3373-447b-9103-15a422c067e9" containerID="cebc6d0ec1d28db86f3a030f98b2b985094b0078402800f06a6d7d015d3e08ee" exitCode=0 Jul 16 12:48:14.843643 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.843541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b12af3b5-3373-447b-9103-15a422c067e9","Type":"ContainerDied","Data":"cebc6d0ec1d28db86f3a030f98b2b985094b0078402800f06a6d7d015d3e08ee"} Jul 16 12:48:14.843643 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.843589 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b12af3b5-3373-447b-9103-15a422c067e9","Type":"ContainerStarted","Data":"8cd154f11111b3e6b5f52f5d24bd19518b7c328bda79ef8dd09e7878f8a4ce00"} Jul 16 12:48:14.986355 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:14.986325 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6a5a92-11c4-4161-9581-f5d33fdad16c" path="/var/lib/kubelet/pods/0f6a5a92-11c4-4161-9581-f5d33fdad16c/volumes" Jul 16 12:48:15.851238 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:15.851200 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b12af3b5-3373-447b-9103-15a422c067e9","Type":"ContainerStarted","Data":"c4baf9299fa2641f324c607aba86da6e8a94fadfe02ba80bbb541c6f7741724e"} Jul 16 12:48:15.851402 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:15.851281 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b12af3b5-3373-447b-9103-15a422c067e9","Type":"ContainerStarted","Data":"6c45e6f0a0b958ba6dcdd4c7b564af9798408c05660a418dff58d8ac02ad60e4"} Jul 16 12:48:15.851402 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:15.851321 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b12af3b5-3373-447b-9103-15a422c067e9","Type":"ContainerStarted","Data":"120b4622b4c2ea9b66477d1b5dc10822eb21777701d7e1923918060a224f074b"} Jul 16 12:48:15.851402 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:15.851360 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b12af3b5-3373-447b-9103-15a422c067e9","Type":"ContainerStarted","Data":"4f13ae8726ecc28156b852ce52ef8a15de74e622bbeb6afe5e3260bd23e4d7a3"} Jul 16 12:48:15.851515 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:15.851402 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b12af3b5-3373-447b-9103-15a422c067e9","Type":"ContainerStarted","Data":"9ad872393ac22b9e1feb72425facf2bb937996ed9f1b2ee5ccc57f9bf6ceedbd"} Jul 16 12:48:15.851515 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:15.851435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b12af3b5-3373-447b-9103-15a422c067e9","Type":"ContainerStarted","Data":"b5fd0d9d22ec2e5d8c0f95d1c78c18b3b75a03e30418a869ad62a98578b8db9c"} Jul 16 12:48:15.891207 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:15.891158 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.87162655 podStartE2EDuration="2.891143314s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.516764ms" isStatefulPod=false podCreationTimestamp="2026-07-16 12:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:48:15.888619134 +0000 UTC m=+181.431225881" watchObservedRunningTime="2026-07-16 12:48:15.891143314 +0000 UTC m=+181.433750120" Jul 16 12:48:19.255557 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:19.255525 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:48:21.561787 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:21.561749 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76df4bc5-4lcd6"] Jul 16 12:48:46.582660 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.582594 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76df4bc5-4lcd6" podUID="93afd687-0e54-4e0c-97cc-5879f7e429a9" containerName="console" containerID="cri-o://79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8" gracePeriod=15 Jul 16 12:48:46.846960 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.846895 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:48:46.938094 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938063 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-config\") pod \"93afd687-0e54-4e0c-97cc-5879f7e429a9\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " Jul 16 12:48:46.938245 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938113 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-serving-cert\") pod \"93afd687-0e54-4e0c-97cc-5879f7e429a9\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " Jul 16 12:48:46.938245 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938147 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-oauth-serving-cert\") pod \"93afd687-0e54-4e0c-97cc-5879f7e429a9\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " Jul 16 12:48:46.938245 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938202 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-oauth-config\") pod \"93afd687-0e54-4e0c-97cc-5879f7e429a9\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " Jul 16 12:48:46.938245 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938226 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-service-ca\") pod \"93afd687-0e54-4e0c-97cc-5879f7e429a9\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " Jul 16 12:48:46.938416 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938289 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-trusted-ca-bundle\") pod \"93afd687-0e54-4e0c-97cc-5879f7e429a9\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " Jul 16 12:48:46.938416 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938315 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm2vk\" (UniqueName: \"kubernetes.io/projected/93afd687-0e54-4e0c-97cc-5879f7e429a9-kube-api-access-jm2vk\") pod \"93afd687-0e54-4e0c-97cc-5879f7e429a9\" (UID: \"93afd687-0e54-4e0c-97cc-5879f7e429a9\") " Jul 16 12:48:46.938550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938518 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-config" (OuterVolumeSpecName: "console-config") pod "93afd687-0e54-4e0c-97cc-5879f7e429a9" (UID: "93afd687-0e54-4e0c-97cc-5879f7e429a9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:46.938705 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938670 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "93afd687-0e54-4e0c-97cc-5879f7e429a9" (UID: "93afd687-0e54-4e0c-97cc-5879f7e429a9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:46.938844 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938721 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-service-ca" (OuterVolumeSpecName: "service-ca") pod "93afd687-0e54-4e0c-97cc-5879f7e429a9" (UID: "93afd687-0e54-4e0c-97cc-5879f7e429a9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:46.938844 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.938739 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "93afd687-0e54-4e0c-97cc-5879f7e429a9" (UID: "93afd687-0e54-4e0c-97cc-5879f7e429a9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 16 12:48:46.940291 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.940268 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "93afd687-0e54-4e0c-97cc-5879f7e429a9" (UID: "93afd687-0e54-4e0c-97cc-5879f7e429a9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:46.940359 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.940286 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "93afd687-0e54-4e0c-97cc-5879f7e429a9" (UID: "93afd687-0e54-4e0c-97cc-5879f7e429a9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:48:46.940451 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.940433 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93afd687-0e54-4e0c-97cc-5879f7e429a9-kube-api-access-jm2vk" (OuterVolumeSpecName: "kube-api-access-jm2vk") pod "93afd687-0e54-4e0c-97cc-5879f7e429a9" (UID: "93afd687-0e54-4e0c-97cc-5879f7e429a9"). InnerVolumeSpecName "kube-api-access-jm2vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:48:46.975012 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.974984 2576 generic.go:356] "Generic (PLEG): container finished" podID="93afd687-0e54-4e0c-97cc-5879f7e429a9" containerID="79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8" exitCode=0 Jul 16 12:48:46.975131 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.975027 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76df4bc5-4lcd6" Jul 16 12:48:46.975131 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.975048 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76df4bc5-4lcd6" event={"ID":"93afd687-0e54-4e0c-97cc-5879f7e429a9","Type":"ContainerDied","Data":"79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8"} Jul 16 12:48:46.975131 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.975111 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76df4bc5-4lcd6" event={"ID":"93afd687-0e54-4e0c-97cc-5879f7e429a9","Type":"ContainerDied","Data":"e519d6165a8398d8ffc486c553b0e7dbd6f5ac44524088a289fc096f8c58e62a"} Jul 16 12:48:46.975131 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.975129 2576 scope.go:117] "RemoveContainer" containerID="79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8" Jul 16 12:48:46.983700 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.983684 2576 scope.go:117] "RemoveContainer" containerID="79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8" Jul 16 12:48:46.983935 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:48:46.983913 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8\": container with ID starting with 79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8 not found: ID does not exist" containerID="79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8" Jul 16 12:48:46.984023 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:46.983938 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8"} err="failed to get container status \"79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8\": rpc error: code = NotFound desc = could not find container \"79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8\": container with ID starting with 79acb28f42976dec207c095c63e498e549ab28a3111ad999ebbbff00cb7accd8 not found: ID does not exist" Jul 16 12:48:47.002328 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.002305 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76df4bc5-4lcd6"] Jul 16 12:48:47.005987 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.005960 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76df4bc5-4lcd6"] Jul 16 12:48:47.039087 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.039067 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-trusted-ca-bundle\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:47.039164 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.039089 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm2vk\" (UniqueName: \"kubernetes.io/projected/93afd687-0e54-4e0c-97cc-5879f7e429a9-kube-api-access-jm2vk\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:47.039164 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.039106 2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-config\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:47.039164 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.039115 2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-serving-cert\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:47.039164 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.039124 2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-oauth-serving-cert\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:47.039164 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.039132 2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/93afd687-0e54-4e0c-97cc-5879f7e429a9-console-oauth-config\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:47.039164 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:47.039140 2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/93afd687-0e54-4e0c-97cc-5879f7e429a9-service-ca\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:48:48.985806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:48:48.985777 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93afd687-0e54-4e0c-97cc-5879f7e429a9" path="/var/lib/kubelet/pods/93afd687-0e54-4e0c-97cc-5879f7e429a9/volumes" Jul 16 12:49:14.255221 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:49:14.255170 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:49:14.270270 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:49:14.270248 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:49:15.093571 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:49:15.093544 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 16 12:50:14.861194 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:14.861162 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 12:50:14.866895 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:14.866871 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 12:50:14.868585 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:14.868569 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 12:50:14.873625 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:14.873571 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 16 12:50:14.876260 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:14.876245 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 12:50:35.444293 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.444237 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb"] Jul 16 12:50:35.446535 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.444649 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93afd687-0e54-4e0c-97cc-5879f7e429a9" containerName="console" Jul 16 12:50:35.446535 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.444666 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="93afd687-0e54-4e0c-97cc-5879f7e429a9" containerName="console" Jul 16 12:50:35.446535 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.444722 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="93afd687-0e54-4e0c-97cc-5879f7e429a9" containerName="console" Jul 16 12:50:35.447516 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.447500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.450655 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.450635 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v8qn5\"" Jul 16 12:50:35.450972 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.450953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 16 12:50:35.451315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.451296 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 16 12:50:35.451398 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.451359 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 16 12:50:35.451677 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.451659 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 16 12:50:35.467236 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.467216 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb"] Jul 16 12:50:35.548648 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.548615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwwkg\" (UniqueName: \"kubernetes.io/projected/ce83b313-fb7f-43fa-94be-9bb05e047ef6-kube-api-access-cwwkg\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.548872 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.548684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce83b313-fb7f-43fa-94be-9bb05e047ef6-tmp\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.548872 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.548767 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce83b313-fb7f-43fa-94be-9bb05e047ef6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.548872 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.548807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce83b313-fb7f-43fa-94be-9bb05e047ef6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.549013 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.548897 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce83b313-fb7f-43fa-94be-9bb05e047ef6-manifests\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.650088 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.650054 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce83b313-fb7f-43fa-94be-9bb05e047ef6-tmp\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.650266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.650104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce83b313-fb7f-43fa-94be-9bb05e047ef6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.650266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.650129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce83b313-fb7f-43fa-94be-9bb05e047ef6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.650266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.650166 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce83b313-fb7f-43fa-94be-9bb05e047ef6-manifests\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.650266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.650205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwwkg\" (UniqueName: \"kubernetes.io/projected/ce83b313-fb7f-43fa-94be-9bb05e047ef6-kube-api-access-cwwkg\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.650604 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.650570 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ce83b313-fb7f-43fa-94be-9bb05e047ef6-manifests\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.652479 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.652451 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce83b313-fb7f-43fa-94be-9bb05e047ef6-tmp\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.652576 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.652545 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ce83b313-fb7f-43fa-94be-9bb05e047ef6-webhook-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.652619 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.652574 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ce83b313-fb7f-43fa-94be-9bb05e047ef6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.659935 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.659913 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwwkg\" (UniqueName: \"kubernetes.io/projected/ce83b313-fb7f-43fa-94be-9bb05e047ef6-kube-api-access-cwwkg\") pod \"opendatahub-operator-controller-manager-9d668944c-dckbb\" (UID: \"ce83b313-fb7f-43fa-94be-9bb05e047ef6\") " pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.760134 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.760059 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:35.898619 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.898595 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb"] Jul 16 12:50:35.900741 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:50:35.900714 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce83b313_fb7f_43fa_94be_9bb05e047ef6.slice/crio-992f1408edc65ac86483d74799d24b90c0a7c809d185e2fd47e841eb6229e5cc WatchSource:0}: Error finding container 992f1408edc65ac86483d74799d24b90c0a7c809d185e2fd47e841eb6229e5cc: Status 404 returned error can't find the container with id 992f1408edc65ac86483d74799d24b90c0a7c809d185e2fd47e841eb6229e5cc Jul 16 12:50:35.902494 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:35.902475 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:50:36.376124 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:36.376091 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" event={"ID":"ce83b313-fb7f-43fa-94be-9bb05e047ef6","Type":"ContainerStarted","Data":"992f1408edc65ac86483d74799d24b90c0a7c809d185e2fd47e841eb6229e5cc"} Jul 16 12:50:39.392721 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:39.392681 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" event={"ID":"ce83b313-fb7f-43fa-94be-9bb05e047ef6","Type":"ContainerStarted","Data":"52f7bfa0f98638275830ce10aad516225d4213eb455f728e39b808fa761154c7"} Jul 16 12:50:40.398550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:40.398459 2576 generic.go:356] "Generic (PLEG): container finished" podID="ce83b313-fb7f-43fa-94be-9bb05e047ef6" containerID="52f7bfa0f98638275830ce10aad516225d4213eb455f728e39b808fa761154c7" exitCode=0 Jul 16 12:50:40.398550 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:40.398506 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" event={"ID":"ce83b313-fb7f-43fa-94be-9bb05e047ef6","Type":"ContainerDied","Data":"52f7bfa0f98638275830ce10aad516225d4213eb455f728e39b808fa761154c7"} Jul 16 12:50:41.404341 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:41.404307 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" event={"ID":"ce83b313-fb7f-43fa-94be-9bb05e047ef6","Type":"ContainerStarted","Data":"a356c7784aef649c9ad0280d0b9994503a86e169cf3acf9194cac3009ff4887a"} Jul 16 12:50:41.404717 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:41.404445 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:41.437711 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:41.437647 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" podStartSLOduration=2.136594252 podStartE2EDuration="6.437625429s" totalImagesPullingTime="3.234283003s" totalInitContainerRuntime="1.066748174s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:50:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:41.43457936 +0000 UTC m=+326.977186134" watchObservedRunningTime="2026-07-16 12:50:41.437625429 +0000 UTC m=+326.980232176" Jul 16 12:50:52.410467 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:52.410440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9d668944c-dckbb" Jul 16 12:50:53.441079 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.441046 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg"] Jul 16 12:50:53.451403 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.451375 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.456224 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.455972 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 16 12:50:53.456224 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.456056 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 16 12:50:53.456224 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.456095 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 16 12:50:53.456224 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.456101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 16 12:50:53.456224 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.456057 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 16 12:50:53.456555 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.456415 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-qwbmj\"" Jul 16 12:50:53.458139 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.458117 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg"] Jul 16 12:50:53.486311 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.486281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/49771ff3-0caf-4106-aa7c-fd31f63bc483-manager-config\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.486430 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.486321 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/49771ff3-0caf-4106-aa7c-fd31f63bc483-metrics-cert\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.486430 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.486395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49771ff3-0caf-4106-aa7c-fd31f63bc483-cert\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.486517 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.486443 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbswn\" (UniqueName: \"kubernetes.io/projected/49771ff3-0caf-4106-aa7c-fd31f63bc483-kube-api-access-gbswn\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.587587 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.587561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/49771ff3-0caf-4106-aa7c-fd31f63bc483-manager-config\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.587740 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.587603 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/49771ff3-0caf-4106-aa7c-fd31f63bc483-metrics-cert\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.587740 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.587668 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49771ff3-0caf-4106-aa7c-fd31f63bc483-cert\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.587740 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.587699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbswn\" (UniqueName: \"kubernetes.io/projected/49771ff3-0caf-4106-aa7c-fd31f63bc483-kube-api-access-gbswn\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.588221 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.588201 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/49771ff3-0caf-4106-aa7c-fd31f63bc483-manager-config\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.590148 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.590124 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/49771ff3-0caf-4106-aa7c-fd31f63bc483-metrics-cert\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.590234 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.590153 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49771ff3-0caf-4106-aa7c-fd31f63bc483-cert\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.597714 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.597691 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbswn\" (UniqueName: \"kubernetes.io/projected/49771ff3-0caf-4106-aa7c-fd31f63bc483-kube-api-access-gbswn\") pod \"lws-controller-manager-5864f4f95b-nfwxg\" (UID: \"49771ff3-0caf-4106-aa7c-fd31f63bc483\") " pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.761487 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.761403 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:53.886012 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:53.885988 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg"] Jul 16 12:50:53.888100 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:50:53.888075 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49771ff3_0caf_4106_aa7c_fd31f63bc483.slice/crio-6c022e5b8eaea9c8b0fb05960384a2adb4ce69da5308dd6ac5d20ecfd98af55f WatchSource:0}: Error finding container 6c022e5b8eaea9c8b0fb05960384a2adb4ce69da5308dd6ac5d20ecfd98af55f: Status 404 returned error can't find the container with id 6c022e5b8eaea9c8b0fb05960384a2adb4ce69da5308dd6ac5d20ecfd98af55f Jul 16 12:50:54.458498 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:54.458452 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" event={"ID":"49771ff3-0caf-4106-aa7c-fd31f63bc483","Type":"ContainerStarted","Data":"6c022e5b8eaea9c8b0fb05960384a2adb4ce69da5308dd6ac5d20ecfd98af55f"} Jul 16 12:50:56.469549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:56.469513 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" event={"ID":"49771ff3-0caf-4106-aa7c-fd31f63bc483","Type":"ContainerStarted","Data":"586c60f607397212fbf1a7951bb6de9e18886256218bb5bd82073e02fd2e98a5"} Jul 16 12:50:56.470000 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:56.469603 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:50:56.510229 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:50:56.510179 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" podStartSLOduration=1.417329183 podStartE2EDuration="3.510164405s" totalImagesPullingTime="2.092835222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:50:56.509025901 +0000 UTC m=+342.051632649" watchObservedRunningTime="2026-07-16 12:50:56.510164405 +0000 UTC m=+342.052771153" Jul 16 12:51:06.759038 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.758959 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg"] Jul 16 12:51:06.762137 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.762121 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.777450 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.777428 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 16 12:51:06.779911 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.779886 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 16 12:51:06.780955 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.780937 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 16 12:51:06.781052 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.780971 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 16 12:51:06.782812 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.782796 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c9sjb\"" Jul 16 12:51:06.800998 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.800973 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg"] Jul 16 12:51:06.803631 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.803610 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3f9b62-2def-4699-bd87-c3bb09c93010-tls-certs\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.803733 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.803658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb5sf\" (UniqueName: \"kubernetes.io/projected/ab3f9b62-2def-4699-bd87-c3bb09c93010-kube-api-access-wb5sf\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.803733 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.803688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab3f9b62-2def-4699-bd87-c3bb09c93010-tmp\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.904632 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.904599 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3f9b62-2def-4699-bd87-c3bb09c93010-tls-certs\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.904796 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.904641 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb5sf\" (UniqueName: \"kubernetes.io/projected/ab3f9b62-2def-4699-bd87-c3bb09c93010-kube-api-access-wb5sf\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.904796 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.904660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab3f9b62-2def-4699-bd87-c3bb09c93010-tmp\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.906933 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.906901 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab3f9b62-2def-4699-bd87-c3bb09c93010-tmp\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.907179 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.907159 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab3f9b62-2def-4699-bd87-c3bb09c93010-tls-certs\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:06.917565 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:06.917538 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb5sf\" (UniqueName: \"kubernetes.io/projected/ab3f9b62-2def-4699-bd87-c3bb09c93010-kube-api-access-wb5sf\") pod \"kube-auth-proxy-d67ccd5b7-5xxlg\" (UID: \"ab3f9b62-2def-4699-bd87-c3bb09c93010\") " pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:07.071411 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:07.071330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" Jul 16 12:51:07.202091 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:07.202062 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg"] Jul 16 12:51:07.204025 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:51:07.203997 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3f9b62_2def_4699_bd87_c3bb09c93010.slice/crio-8db2dd93643a56cd08b22a5a4df75ace40ed5b60f4c740f7b4ddf5da506cb559 WatchSource:0}: Error finding container 8db2dd93643a56cd08b22a5a4df75ace40ed5b60f4c740f7b4ddf5da506cb559: Status 404 returned error can't find the container with id 8db2dd93643a56cd08b22a5a4df75ace40ed5b60f4c740f7b4ddf5da506cb559 Jul 16 12:51:07.476324 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:07.476300 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5864f4f95b-nfwxg" Jul 16 12:51:07.515665 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:07.515635 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" event={"ID":"ab3f9b62-2def-4699-bd87-c3bb09c93010","Type":"ContainerStarted","Data":"8db2dd93643a56cd08b22a5a4df75ace40ed5b60f4c740f7b4ddf5da506cb559"} Jul 16 12:51:10.536126 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:10.536090 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" event={"ID":"ab3f9b62-2def-4699-bd87-c3bb09c93010","Type":"ContainerStarted","Data":"1dfb13a2c0d7473d096bec41ad1bae1c7ef7f8ff49c1b32f6309d3b393123b77"} Jul 16 12:51:10.561456 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:51:10.561411 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d67ccd5b7-5xxlg" podStartSLOduration=1.307758103 podStartE2EDuration="4.561397516s" totalImagesPullingTime="3.253639413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:51:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:51:10.559638111 +0000 UTC m=+356.102244858" watchObservedRunningTime="2026-07-16 12:51:10.561397516 +0000 UTC m=+356.104004262" Jul 16 12:52:43.843594 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.843514 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l"] Jul 16 12:52:43.846274 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.846259 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:43.849536 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.849511 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 16 12:52:43.849657 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.849536 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 16 12:52:43.850188 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.850169 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 16 12:52:43.850307 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.850196 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 16 12:52:43.850307 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.850288 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-97zh8\"" Jul 16 12:52:43.860380 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.860361 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l"] Jul 16 12:52:43.904060 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.904034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b633b4c9-328f-4b8c-8be6-f7d79c481c66-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:43.904185 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.904112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qld77\" (UniqueName: \"kubernetes.io/projected/b633b4c9-328f-4b8c-8be6-f7d79c481c66-kube-api-access-qld77\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:43.904246 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:43.904178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:44.005001 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:44.004971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qld77\" (UniqueName: \"kubernetes.io/projected/b633b4c9-328f-4b8c-8be6-f7d79c481c66-kube-api-access-qld77\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:44.005164 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:44.005015 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:44.005164 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:44.005063 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b633b4c9-328f-4b8c-8be6-f7d79c481c66-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:44.005285 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:52:44.005198 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 16 12:52:44.005285 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:52:44.005277 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert podName:b633b4c9-328f-4b8c-8be6-f7d79c481c66 nodeName:}" failed. No retries permitted until 2026-07-16 12:52:44.505255427 +0000 UTC m=+450.047862169 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-w8g8l" (UID: "b633b4c9-328f-4b8c-8be6-f7d79c481c66") : secret "plugin-serving-cert" not found Jul 16 12:52:44.005604 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:44.005586 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b633b4c9-328f-4b8c-8be6-f7d79c481c66-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:44.016323 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:44.016297 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qld77\" (UniqueName: \"kubernetes.io/projected/b633b4c9-328f-4b8c-8be6-f7d79c481c66-kube-api-access-qld77\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:44.508245 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:44.508214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:44.508429 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:52:44.508372 2576 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 16 12:52:44.508470 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:52:44.508450 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert podName:b633b4c9-328f-4b8c-8be6-f7d79c481c66 nodeName:}" failed. No retries permitted until 2026-07-16 12:52:45.508432584 +0000 UTC m=+451.051039311 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-w8g8l" (UID: "b633b4c9-328f-4b8c-8be6-f7d79c481c66") : secret "plugin-serving-cert" not found Jul 16 12:52:45.515996 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:45.515961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:45.518448 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:45.518429 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b633b4c9-328f-4b8c-8be6-f7d79c481c66-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-w8g8l\" (UID: \"b633b4c9-328f-4b8c-8be6-f7d79c481c66\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:45.665667 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:45.665641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" Jul 16 12:52:45.803806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:45.803722 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l"] Jul 16 12:52:45.806564 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:52:45.806540 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb633b4c9_328f_4b8c_8be6_f7d79c481c66.slice/crio-a038ffe79689e3837a0fe919bf85e215e37743a2a4c97f9c0a716ad021163f09 WatchSource:0}: Error finding container a038ffe79689e3837a0fe919bf85e215e37743a2a4c97f9c0a716ad021163f09: Status 404 returned error can't find the container with id a038ffe79689e3837a0fe919bf85e215e37743a2a4c97f9c0a716ad021163f09 Jul 16 12:52:45.935355 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:45.935322 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" event={"ID":"b633b4c9-328f-4b8c-8be6-f7d79c481c66","Type":"ContainerStarted","Data":"a038ffe79689e3837a0fe919bf85e215e37743a2a4c97f9c0a716ad021163f09"} Jul 16 12:52:54.055181 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.055144 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg"] Jul 16 12:52:54.065153 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.065131 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:52:54.068482 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.068457 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fh5x6\"" Jul 16 12:52:54.080447 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.080427 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg"] Jul 16 12:52:54.104069 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.104027 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7491ba53-355d-4992-9e7c-63ac42ae45d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" (UID: \"7491ba53-355d-4992-9e7c-63ac42ae45d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:52:54.104213 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.104131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62dg\" (UniqueName: \"kubernetes.io/projected/7491ba53-355d-4992-9e7c-63ac42ae45d3-kube-api-access-n62dg\") pod \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" (UID: \"7491ba53-355d-4992-9e7c-63ac42ae45d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:52:54.204718 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.204682 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62dg\" (UniqueName: \"kubernetes.io/projected/7491ba53-355d-4992-9e7c-63ac42ae45d3-kube-api-access-n62dg\") pod \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" (UID: \"7491ba53-355d-4992-9e7c-63ac42ae45d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:52:54.204931 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.204797 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7491ba53-355d-4992-9e7c-63ac42ae45d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" (UID: \"7491ba53-355d-4992-9e7c-63ac42ae45d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:52:54.205152 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.205134 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7491ba53-355d-4992-9e7c-63ac42ae45d3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" (UID: \"7491ba53-355d-4992-9e7c-63ac42ae45d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:52:54.231864 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.231831 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62dg\" (UniqueName: \"kubernetes.io/projected/7491ba53-355d-4992-9e7c-63ac42ae45d3-kube-api-access-n62dg\") pod \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" (UID: \"7491ba53-355d-4992-9e7c-63ac42ae45d3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:52:54.378156 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.378082 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:52:54.603798 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.603763 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg"] Jul 16 12:52:54.610532 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.610474 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg"] Jul 16 12:52:54.630642 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.630569 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf"] Jul 16 12:52:54.637496 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.637474 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:52:54.657052 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.656919 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf"] Jul 16 12:52:54.709953 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.709924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82498410-42c7-4517-9399-417008f3ed9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6h8pf\" (UID: \"82498410-42c7-4517-9399-417008f3ed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:52:54.710131 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.710044 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8np7\" (UniqueName: \"kubernetes.io/projected/82498410-42c7-4517-9399-417008f3ed9e-kube-api-access-w8np7\") pod \"kuadrant-operator-controller-manager-55467d9f7-6h8pf\" (UID: \"82498410-42c7-4517-9399-417008f3ed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:52:54.811187 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.811149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8np7\" (UniqueName: \"kubernetes.io/projected/82498410-42c7-4517-9399-417008f3ed9e-kube-api-access-w8np7\") pod \"kuadrant-operator-controller-manager-55467d9f7-6h8pf\" (UID: \"82498410-42c7-4517-9399-417008f3ed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:52:54.811351 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.811247 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82498410-42c7-4517-9399-417008f3ed9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6h8pf\" (UID: \"82498410-42c7-4517-9399-417008f3ed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:52:54.811652 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.811631 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82498410-42c7-4517-9399-417008f3ed9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-6h8pf\" (UID: \"82498410-42c7-4517-9399-417008f3ed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:52:54.822126 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.822097 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8np7\" (UniqueName: \"kubernetes.io/projected/82498410-42c7-4517-9399-417008f3ed9e-kube-api-access-w8np7\") pod \"kuadrant-operator-controller-manager-55467d9f7-6h8pf\" (UID: \"82498410-42c7-4517-9399-417008f3ed9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:52:54.951534 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:52:54.951493 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:53:06.672038 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:06.672015 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf"] Jul 16 12:53:06.673436 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:53:06.673411 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82498410_42c7_4517_9399_417008f3ed9e.slice/crio-6c02992b3d395677a7e19e957e7cc10fd7af8bf034ed583a88c3d1ae64b7ccf0 WatchSource:0}: Error finding container 6c02992b3d395677a7e19e957e7cc10fd7af8bf034ed583a88c3d1ae64b7ccf0: Status 404 returned error can't find the container with id 6c02992b3d395677a7e19e957e7cc10fd7af8bf034ed583a88c3d1ae64b7ccf0 Jul 16 12:53:07.046252 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:07.046201 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" event={"ID":"b633b4c9-328f-4b8c-8be6-f7d79c481c66","Type":"ContainerStarted","Data":"24005e3b6c17f6ddb17dc68230c71f25e704103408e4d28e5850471cf57653d1"} Jul 16 12:53:07.047421 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:07.047395 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" event={"ID":"82498410-42c7-4517-9399-417008f3ed9e","Type":"ContainerStarted","Data":"6c02992b3d395677a7e19e957e7cc10fd7af8bf034ed583a88c3d1ae64b7ccf0"} Jul 16 12:53:07.066604 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:07.066562 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-w8g8l" podStartSLOduration=3.296750223 podStartE2EDuration="24.066547547s" totalImagesPullingTime="20.769797324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:52:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:53:07.065483976 +0000 UTC m=+472.608090724" watchObservedRunningTime="2026-07-16 12:53:07.066547547 +0000 UTC m=+472.609154294" Jul 16 12:53:09.065573 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:53:09.065529 2576 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-ngzbg_kuadrant-system_7491ba53-355d-4992-9e7c-63ac42ae45d3_0(f15de6ce7132a7a00e032b49e9e0adbfdbb31f7be00fe1713a8e21353c0c03b5): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-ngzbg to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 16 12:53:09.065998 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:53:09.065600 2576 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-ngzbg_kuadrant-system_7491ba53-355d-4992-9e7c-63ac42ae45d3_0(f15de6ce7132a7a00e032b49e9e0adbfdbb31f7be00fe1713a8e21353c0c03b5): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-ngzbg to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:53:10.062011 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.061981 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:53:10.064896 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:53:10.064861 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="7491ba53-355d-4992-9e7c-63ac42ae45d3" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:53:10.087428 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.087399 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:53:10.090256 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:53:10.090214 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="7491ba53-355d-4992-9e7c-63ac42ae45d3" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:53:10.257430 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.257397 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n62dg\" (UniqueName: \"kubernetes.io/projected/7491ba53-355d-4992-9e7c-63ac42ae45d3-kube-api-access-n62dg\") pod \"7491ba53-355d-4992-9e7c-63ac42ae45d3\" (UID: \"7491ba53-355d-4992-9e7c-63ac42ae45d3\") " Jul 16 12:53:10.257623 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.257465 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7491ba53-355d-4992-9e7c-63ac42ae45d3-extensions-socket-volume\") pod \"7491ba53-355d-4992-9e7c-63ac42ae45d3\" (UID: \"7491ba53-355d-4992-9e7c-63ac42ae45d3\") " Jul 16 12:53:10.257723 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.257700 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7491ba53-355d-4992-9e7c-63ac42ae45d3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7491ba53-355d-4992-9e7c-63ac42ae45d3" (UID: "7491ba53-355d-4992-9e7c-63ac42ae45d3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:53:10.259349 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.259325 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7491ba53-355d-4992-9e7c-63ac42ae45d3-kube-api-access-n62dg" (OuterVolumeSpecName: "kube-api-access-n62dg") pod "7491ba53-355d-4992-9e7c-63ac42ae45d3" (UID: "7491ba53-355d-4992-9e7c-63ac42ae45d3"). InnerVolumeSpecName "kube-api-access-n62dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:53:10.358499 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.358429 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7491ba53-355d-4992-9e7c-63ac42ae45d3-extensions-socket-volume\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:53:10.358499 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.358455 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62dg\" (UniqueName: \"kubernetes.io/projected/7491ba53-355d-4992-9e7c-63ac42ae45d3-kube-api-access-n62dg\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:53:10.988308 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:10.988268 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7491ba53-355d-4992-9e7c-63ac42ae45d3" path="/var/lib/kubelet/pods/7491ba53-355d-4992-9e7c-63ac42ae45d3/volumes" Jul 16 12:53:11.066313 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:11.066286 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:53:11.069162 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:53:11.069129 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="7491ba53-355d-4992-9e7c-63ac42ae45d3" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:53:11.072223 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:53:11.072188 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-ngzbg\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="7491ba53-355d-4992-9e7c-63ac42ae45d3" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ngzbg" Jul 16 12:53:14.085899 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:14.085865 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" event={"ID":"82498410-42c7-4517-9399-417008f3ed9e","Type":"ContainerStarted","Data":"d4b8dc8c5c250e82f4add216acdee2d84991cfb17dd898bb33e2dd9453e5b0a7"} Jul 16 12:53:14.086399 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:14.085946 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:53:14.113652 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:14.113599 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" podStartSLOduration=13.48762021 podStartE2EDuration="20.113585841s" totalImagesPullingTime="6.625965631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:53:14.110765418 +0000 UTC m=+479.653372199" watchObservedRunningTime="2026-07-16 12:53:14.113585841 +0000 UTC m=+479.656192587" Jul 16 12:53:25.092945 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:25.092916 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:53:37.944905 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:37.944871 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf"] Jul 16 12:53:37.945315 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:37.945063 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" podUID="82498410-42c7-4517-9399-417008f3ed9e" containerName="manager" containerID="cri-o://d4b8dc8c5c250e82f4add216acdee2d84991cfb17dd898bb33e2dd9453e5b0a7" gracePeriod=10 Jul 16 12:53:38.188083 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.188046 2576 generic.go:356] "Generic (PLEG): container finished" podID="82498410-42c7-4517-9399-417008f3ed9e" containerID="d4b8dc8c5c250e82f4add216acdee2d84991cfb17dd898bb33e2dd9453e5b0a7" exitCode=0 Jul 16 12:53:38.188205 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.188120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" event={"ID":"82498410-42c7-4517-9399-417008f3ed9e","Type":"ContainerDied","Data":"d4b8dc8c5c250e82f4add216acdee2d84991cfb17dd898bb33e2dd9453e5b0a7"} Jul 16 12:53:38.188267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.188213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" event={"ID":"82498410-42c7-4517-9399-417008f3ed9e","Type":"ContainerDied","Data":"6c02992b3d395677a7e19e957e7cc10fd7af8bf034ed583a88c3d1ae64b7ccf0"} Jul 16 12:53:38.188267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.188239 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c02992b3d395677a7e19e957e7cc10fd7af8bf034ed583a88c3d1ae64b7ccf0" Jul 16 12:53:38.207600 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.207540 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:53:38.313125 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.313097 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8np7\" (UniqueName: \"kubernetes.io/projected/82498410-42c7-4517-9399-417008f3ed9e-kube-api-access-w8np7\") pod \"82498410-42c7-4517-9399-417008f3ed9e\" (UID: \"82498410-42c7-4517-9399-417008f3ed9e\") " Jul 16 12:53:38.313273 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.313164 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82498410-42c7-4517-9399-417008f3ed9e-extensions-socket-volume\") pod \"82498410-42c7-4517-9399-417008f3ed9e\" (UID: \"82498410-42c7-4517-9399-417008f3ed9e\") " Jul 16 12:53:38.313542 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.313523 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82498410-42c7-4517-9399-417008f3ed9e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "82498410-42c7-4517-9399-417008f3ed9e" (UID: "82498410-42c7-4517-9399-417008f3ed9e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 12:53:38.315404 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.315379 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82498410-42c7-4517-9399-417008f3ed9e-kube-api-access-w8np7" (OuterVolumeSpecName: "kube-api-access-w8np7") pod "82498410-42c7-4517-9399-417008f3ed9e" (UID: "82498410-42c7-4517-9399-417008f3ed9e"). InnerVolumeSpecName "kube-api-access-w8np7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:53:38.414706 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.414663 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82498410-42c7-4517-9399-417008f3ed9e-extensions-socket-volume\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:53:38.414706 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:38.414700 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8np7\" (UniqueName: \"kubernetes.io/projected/82498410-42c7-4517-9399-417008f3ed9e-kube-api-access-w8np7\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:53:39.193494 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:39.193461 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf" Jul 16 12:53:39.216693 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:39.216669 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf"] Jul 16 12:53:39.231333 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:39.231296 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-6h8pf"] Jul 16 12:53:40.986424 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:40.986391 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82498410-42c7-4517-9399-417008f3ed9e" path="/var/lib/kubelet/pods/82498410-42c7-4517-9399-417008f3ed9e/volumes" Jul 16 12:53:44.654001 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.653967 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:53:44.654407 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.654334 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82498410-42c7-4517-9399-417008f3ed9e" containerName="manager" Jul 16 12:53:44.654407 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.654346 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82498410-42c7-4517-9399-417008f3ed9e" containerName="manager" Jul 16 12:53:44.654407 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.654397 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82498410-42c7-4517-9399-417008f3ed9e" containerName="manager" Jul 16 12:53:44.688712 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.688679 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:53:44.688870 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.688848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:44.691319 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.691298 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:53:44.692660 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.692638 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 16 12:53:44.768958 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.768928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2hnc\" (UniqueName: \"kubernetes.io/projected/b36329e8-ad97-46cd-978e-4dd3f86de5a6-kube-api-access-m2hnc\") pod \"limitador-limitador-54d6d988d5-xsdkx\" (UID: \"b36329e8-ad97-46cd-978e-4dd3f86de5a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:44.768958 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.768963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b36329e8-ad97-46cd-978e-4dd3f86de5a6-config-file\") pod \"limitador-limitador-54d6d988d5-xsdkx\" (UID: \"b36329e8-ad97-46cd-978e-4dd3f86de5a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:44.870238 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.870201 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2hnc\" (UniqueName: \"kubernetes.io/projected/b36329e8-ad97-46cd-978e-4dd3f86de5a6-kube-api-access-m2hnc\") pod \"limitador-limitador-54d6d988d5-xsdkx\" (UID: \"b36329e8-ad97-46cd-978e-4dd3f86de5a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:44.870238 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.870240 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b36329e8-ad97-46cd-978e-4dd3f86de5a6-config-file\") pod \"limitador-limitador-54d6d988d5-xsdkx\" (UID: \"b36329e8-ad97-46cd-978e-4dd3f86de5a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:44.870804 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.870786 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b36329e8-ad97-46cd-978e-4dd3f86de5a6-config-file\") pod \"limitador-limitador-54d6d988d5-xsdkx\" (UID: \"b36329e8-ad97-46cd-978e-4dd3f86de5a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:44.880678 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.880653 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2hnc\" (UniqueName: \"kubernetes.io/projected/b36329e8-ad97-46cd-978e-4dd3f86de5a6-kube-api-access-m2hnc\") pod \"limitador-limitador-54d6d988d5-xsdkx\" (UID: \"b36329e8-ad97-46cd-978e-4dd3f86de5a6\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:44.998952 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:44.998916 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:45.158305 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:45.158253 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:53:45.161698 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:53:45.161662 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36329e8_ad97_46cd_978e_4dd3f86de5a6.slice/crio-b7785f56b6e9eb33e5026927968c7a1b2690dcc7b30f39acb77016ff51281703 WatchSource:0}: Error finding container b7785f56b6e9eb33e5026927968c7a1b2690dcc7b30f39acb77016ff51281703: Status 404 returned error can't find the container with id b7785f56b6e9eb33e5026927968c7a1b2690dcc7b30f39acb77016ff51281703 Jul 16 12:53:45.221047 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:45.221016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" event={"ID":"b36329e8-ad97-46cd-978e-4dd3f86de5a6","Type":"ContainerStarted","Data":"b7785f56b6e9eb33e5026927968c7a1b2690dcc7b30f39acb77016ff51281703"} Jul 16 12:53:45.843987 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:45.843955 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-khp4k"] Jul 16 12:53:45.865335 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:45.865307 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-khp4k"] Jul 16 12:53:45.865512 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:45.865416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-khp4k" Jul 16 12:53:45.868962 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:45.868909 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xpgq8\"" Jul 16 12:53:45.981193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:45.981157 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7s8\" (UniqueName: \"kubernetes.io/projected/e17ce232-ae35-452b-985b-e4c19ea3808e-kube-api-access-hk7s8\") pod \"authorino-75d556bd77-khp4k\" (UID: \"e17ce232-ae35-452b-985b-e4c19ea3808e\") " pod="kuadrant-system/authorino-75d556bd77-khp4k" Jul 16 12:53:46.081933 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:46.081903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7s8\" (UniqueName: \"kubernetes.io/projected/e17ce232-ae35-452b-985b-e4c19ea3808e-kube-api-access-hk7s8\") pod \"authorino-75d556bd77-khp4k\" (UID: \"e17ce232-ae35-452b-985b-e4c19ea3808e\") " pod="kuadrant-system/authorino-75d556bd77-khp4k" Jul 16 12:53:46.091316 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:46.091292 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7s8\" (UniqueName: \"kubernetes.io/projected/e17ce232-ae35-452b-985b-e4c19ea3808e-kube-api-access-hk7s8\") pod \"authorino-75d556bd77-khp4k\" (UID: \"e17ce232-ae35-452b-985b-e4c19ea3808e\") " pod="kuadrant-system/authorino-75d556bd77-khp4k" Jul 16 12:53:46.177778 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:46.177743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-khp4k" Jul 16 12:53:46.351334 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:46.351275 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-khp4k"] Jul 16 12:53:46.354612 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:53:46.354583 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode17ce232_ae35_452b_985b_e4c19ea3808e.slice/crio-4acc38ff908755b49c4fb73e6b4e1412021593667d70c88c25dfa6912805222d WatchSource:0}: Error finding container 4acc38ff908755b49c4fb73e6b4e1412021593667d70c88c25dfa6912805222d: Status 404 returned error can't find the container with id 4acc38ff908755b49c4fb73e6b4e1412021593667d70c88c25dfa6912805222d Jul 16 12:53:47.238323 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:47.238285 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-khp4k" event={"ID":"e17ce232-ae35-452b-985b-e4c19ea3808e","Type":"ContainerStarted","Data":"4acc38ff908755b49c4fb73e6b4e1412021593667d70c88c25dfa6912805222d"} Jul 16 12:53:50.259900 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:50.259862 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-khp4k" event={"ID":"e17ce232-ae35-452b-985b-e4c19ea3808e","Type":"ContainerStarted","Data":"d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711"} Jul 16 12:53:50.261445 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:50.261421 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" event={"ID":"b36329e8-ad97-46cd-978e-4dd3f86de5a6","Type":"ContainerStarted","Data":"e3882fa518efed9111b8ab0861a482f04b11c34faf743d2720b0f882fd75b5cf"} Jul 16 12:53:50.261570 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:50.261494 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:53:50.286796 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:53:50.286755 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-khp4k" podStartSLOduration=1.597799668 podStartE2EDuration="5.286742252s" totalImagesPullingTime="3.688942584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:53:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:53:50.284553696 +0000 UTC m=+515.827160444" watchObservedRunningTime="2026-07-16 12:53:50.286742252 +0000 UTC m=+515.829348996" Jul 16 12:54:01.268298 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:01.268266 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" Jul 16 12:54:01.295194 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:01.295143 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-xsdkx" podStartSLOduration=12.41484935 podStartE2EDuration="17.295085031s" totalImagesPullingTime="4.880235681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:53:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:53:50.328786961 +0000 UTC m=+515.871393708" watchObservedRunningTime="2026-07-16 12:54:01.295085031 +0000 UTC m=+526.837691778" Jul 16 12:54:18.756251 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:18.756170 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xbjcn"] Jul 16 12:54:18.762361 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:18.762337 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" Jul 16 12:54:18.782176 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:18.782154 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xbjcn"] Jul 16 12:54:18.872425 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:18.872390 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjprg\" (UniqueName: \"kubernetes.io/projected/e1f29a63-5601-4894-993e-161c57beae15-kube-api-access-qjprg\") pod \"authorino-5d4b67c446-xbjcn\" (UID: \"e1f29a63-5601-4894-993e-161c57beae15\") " pod="kuadrant-system/authorino-5d4b67c446-xbjcn" Jul 16 12:54:18.973333 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:18.973303 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjprg\" (UniqueName: \"kubernetes.io/projected/e1f29a63-5601-4894-993e-161c57beae15-kube-api-access-qjprg\") pod \"authorino-5d4b67c446-xbjcn\" (UID: \"e1f29a63-5601-4894-993e-161c57beae15\") " pod="kuadrant-system/authorino-5d4b67c446-xbjcn" Jul 16 12:54:18.991770 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:18.991740 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjprg\" (UniqueName: \"kubernetes.io/projected/e1f29a63-5601-4894-993e-161c57beae15-kube-api-access-qjprg\") pod \"authorino-5d4b67c446-xbjcn\" (UID: \"e1f29a63-5601-4894-993e-161c57beae15\") " pod="kuadrant-system/authorino-5d4b67c446-xbjcn" Jul 16 12:54:19.008062 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.008002 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xbjcn"] Jul 16 12:54:19.008220 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.008210 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" Jul 16 12:54:19.046099 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.046064 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7fcc644988-gk6dp"] Jul 16 12:54:19.050668 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.050647 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fcc644988-gk6dp" Jul 16 12:54:19.063017 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.062989 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7fcc644988-gk6dp"] Jul 16 12:54:19.150309 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.150285 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xbjcn"] Jul 16 12:54:19.152608 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:54:19.152580 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1f29a63_5601_4894_993e_161c57beae15.slice/crio-8af9d1d93080b7edc296ec0b07d8e2a2f23828ebcd043a7554904d30c79f7edf WatchSource:0}: Error finding container 8af9d1d93080b7edc296ec0b07d8e2a2f23828ebcd043a7554904d30c79f7edf: Status 404 returned error can't find the container with id 8af9d1d93080b7edc296ec0b07d8e2a2f23828ebcd043a7554904d30c79f7edf Jul 16 12:54:19.175464 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.175431 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czlxx\" (UniqueName: \"kubernetes.io/projected/501dd6ff-3955-459d-aaa4-16016377db50-kube-api-access-czlxx\") pod \"authorino-7fcc644988-gk6dp\" (UID: \"501dd6ff-3955-459d-aaa4-16016377db50\") " pod="kuadrant-system/authorino-7fcc644988-gk6dp" Jul 16 12:54:19.276504 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.276432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czlxx\" (UniqueName: \"kubernetes.io/projected/501dd6ff-3955-459d-aaa4-16016377db50-kube-api-access-czlxx\") pod \"authorino-7fcc644988-gk6dp\" (UID: \"501dd6ff-3955-459d-aaa4-16016377db50\") " pod="kuadrant-system/authorino-7fcc644988-gk6dp" Jul 16 12:54:19.291933 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.291911 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czlxx\" (UniqueName: \"kubernetes.io/projected/501dd6ff-3955-459d-aaa4-16016377db50-kube-api-access-czlxx\") pod \"authorino-7fcc644988-gk6dp\" (UID: \"501dd6ff-3955-459d-aaa4-16016377db50\") " pod="kuadrant-system/authorino-7fcc644988-gk6dp" Jul 16 12:54:19.360125 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.360098 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fcc644988-gk6dp"] Jul 16 12:54:19.360322 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.360311 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fcc644988-gk6dp" Jul 16 12:54:19.386956 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.386921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" event={"ID":"e1f29a63-5601-4894-993e-161c57beae15","Type":"ContainerStarted","Data":"8af9d1d93080b7edc296ec0b07d8e2a2f23828ebcd043a7554904d30c79f7edf"} Jul 16 12:54:19.405517 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.405486 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-845fdc676d-t8mnc"] Jul 16 12:54:19.410720 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.410685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:54:19.414493 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.414300 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 16 12:54:19.422489 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.422464 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-845fdc676d-t8mnc"] Jul 16 12:54:19.477587 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.477558 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59da6f90-846a-4ce9-8694-6aa441962a31-tls-cert\") pod \"authorino-845fdc676d-t8mnc\" (UID: \"59da6f90-846a-4ce9-8694-6aa441962a31\") " pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:54:19.477760 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.477607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qhr9\" (UniqueName: \"kubernetes.io/projected/59da6f90-846a-4ce9-8694-6aa441962a31-kube-api-access-9qhr9\") pod \"authorino-845fdc676d-t8mnc\" (UID: \"59da6f90-846a-4ce9-8694-6aa441962a31\") " pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:54:19.496469 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.496448 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fcc644988-gk6dp"] Jul 16 12:54:19.497791 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:54:19.497769 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod501dd6ff_3955_459d_aaa4_16016377db50.slice/crio-a28dc89b17d57a183d1d947e48c448fab8119537fba05b4bf2ed1b9d6284643e WatchSource:0}: Error finding container a28dc89b17d57a183d1d947e48c448fab8119537fba05b4bf2ed1b9d6284643e: Status 404 returned error can't find the container with id a28dc89b17d57a183d1d947e48c448fab8119537fba05b4bf2ed1b9d6284643e Jul 16 12:54:19.578715 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.578687 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qhr9\" (UniqueName: \"kubernetes.io/projected/59da6f90-846a-4ce9-8694-6aa441962a31-kube-api-access-9qhr9\") pod \"authorino-845fdc676d-t8mnc\" (UID: \"59da6f90-846a-4ce9-8694-6aa441962a31\") " pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:54:19.578922 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.578833 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59da6f90-846a-4ce9-8694-6aa441962a31-tls-cert\") pod \"authorino-845fdc676d-t8mnc\" (UID: \"59da6f90-846a-4ce9-8694-6aa441962a31\") " pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:54:19.581058 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.581039 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59da6f90-846a-4ce9-8694-6aa441962a31-tls-cert\") pod \"authorino-845fdc676d-t8mnc\" (UID: \"59da6f90-846a-4ce9-8694-6aa441962a31\") " pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:54:19.591215 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.591173 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qhr9\" (UniqueName: \"kubernetes.io/projected/59da6f90-846a-4ce9-8694-6aa441962a31-kube-api-access-9qhr9\") pod \"authorino-845fdc676d-t8mnc\" (UID: \"59da6f90-846a-4ce9-8694-6aa441962a31\") " pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:54:19.722831 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.722784 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:54:19.868867 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:19.868841 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-845fdc676d-t8mnc"] Jul 16 12:54:19.871732 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:54:19.871710 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59da6f90_846a_4ce9_8694_6aa441962a31.slice/crio-11147faed370608658b66a75ee96da7a5dcfb44cf74a6c499589656950b76944 WatchSource:0}: Error finding container 11147faed370608658b66a75ee96da7a5dcfb44cf74a6c499589656950b76944: Status 404 returned error can't find the container with id 11147faed370608658b66a75ee96da7a5dcfb44cf74a6c499589656950b76944 Jul 16 12:54:20.396001 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.395965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fcc644988-gk6dp" event={"ID":"501dd6ff-3955-459d-aaa4-16016377db50","Type":"ContainerStarted","Data":"85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba"} Jul 16 12:54:20.396001 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.395988 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7fcc644988-gk6dp" podUID="501dd6ff-3955-459d-aaa4-16016377db50" containerName="authorino" containerID="cri-o://85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba" gracePeriod=30 Jul 16 12:54:20.396232 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.396009 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fcc644988-gk6dp" event={"ID":"501dd6ff-3955-459d-aaa4-16016377db50","Type":"ContainerStarted","Data":"a28dc89b17d57a183d1d947e48c448fab8119537fba05b4bf2ed1b9d6284643e"} Jul 16 12:54:20.397270 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.397234 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845fdc676d-t8mnc" event={"ID":"59da6f90-846a-4ce9-8694-6aa441962a31","Type":"ContainerStarted","Data":"11147faed370608658b66a75ee96da7a5dcfb44cf74a6c499589656950b76944"} Jul 16 12:54:20.398597 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.398563 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" event={"ID":"e1f29a63-5601-4894-993e-161c57beae15","Type":"ContainerStarted","Data":"de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b"} Jul 16 12:54:20.398688 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.398635 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" podUID="e1f29a63-5601-4894-993e-161c57beae15" containerName="authorino" containerID="cri-o://de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b" gracePeriod=30 Jul 16 12:54:20.419582 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.419540 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7fcc644988-gk6dp" podStartSLOduration=1.084710542 podStartE2EDuration="1.419527016s" totalImagesPullingTime="334.816474ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:54:20.417280894 +0000 UTC m=+545.959887641" watchObservedRunningTime="2026-07-16 12:54:20.419527016 +0000 UTC m=+545.962133810" Jul 16 12:54:20.438278 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.438245 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" podStartSLOduration=2.10591133 podStartE2EDuration="2.438232892s" totalImagesPullingTime="332.321562ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:54:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:54:20.437339239 +0000 UTC m=+545.979945985" watchObservedRunningTime="2026-07-16 12:54:20.438232892 +0000 UTC m=+545.980839638" Jul 16 12:54:20.649954 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.649894 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fcc644988-gk6dp" Jul 16 12:54:20.652770 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.652750 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" Jul 16 12:54:20.794549 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.794516 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qjprg\" (UniqueName: \"kubernetes.io/projected/e1f29a63-5601-4894-993e-161c57beae15-kube-api-access-qjprg\") pod \"e1f29a63-5601-4894-993e-161c57beae15\" (UID: \"e1f29a63-5601-4894-993e-161c57beae15\") " Jul 16 12:54:20.794725 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.794572 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czlxx\" (UniqueName: \"kubernetes.io/projected/501dd6ff-3955-459d-aaa4-16016377db50-kube-api-access-czlxx\") pod \"501dd6ff-3955-459d-aaa4-16016377db50\" (UID: \"501dd6ff-3955-459d-aaa4-16016377db50\") " Jul 16 12:54:20.796539 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.796513 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/501dd6ff-3955-459d-aaa4-16016377db50-kube-api-access-czlxx" (OuterVolumeSpecName: "kube-api-access-czlxx") pod "501dd6ff-3955-459d-aaa4-16016377db50" (UID: "501dd6ff-3955-459d-aaa4-16016377db50"). InnerVolumeSpecName "kube-api-access-czlxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:54:20.796634 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.796550 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f29a63-5601-4894-993e-161c57beae15-kube-api-access-qjprg" (OuterVolumeSpecName: "kube-api-access-qjprg") pod "e1f29a63-5601-4894-993e-161c57beae15" (UID: "e1f29a63-5601-4894-993e-161c57beae15"). InnerVolumeSpecName "kube-api-access-qjprg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:54:20.895291 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.895260 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjprg\" (UniqueName: \"kubernetes.io/projected/e1f29a63-5601-4894-993e-161c57beae15-kube-api-access-qjprg\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:54:20.895291 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:20.895286 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czlxx\" (UniqueName: \"kubernetes.io/projected/501dd6ff-3955-459d-aaa4-16016377db50-kube-api-access-czlxx\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:54:21.405144 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.405061 2576 generic.go:356] "Generic (PLEG): container finished" podID="501dd6ff-3955-459d-aaa4-16016377db50" containerID="85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba" exitCode=0 Jul 16 12:54:21.405144 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.405104 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7fcc644988-gk6dp" Jul 16 12:54:21.405369 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.405103 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fcc644988-gk6dp" event={"ID":"501dd6ff-3955-459d-aaa4-16016377db50","Type":"ContainerDied","Data":"85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba"} Jul 16 12:54:21.405369 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.405220 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7fcc644988-gk6dp" event={"ID":"501dd6ff-3955-459d-aaa4-16016377db50","Type":"ContainerDied","Data":"a28dc89b17d57a183d1d947e48c448fab8119537fba05b4bf2ed1b9d6284643e"} Jul 16 12:54:21.405369 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.405261 2576 scope.go:117] "RemoveContainer" containerID="85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba" Jul 16 12:54:21.406927 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.406900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845fdc676d-t8mnc" event={"ID":"59da6f90-846a-4ce9-8694-6aa441962a31","Type":"ContainerStarted","Data":"a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe"} Jul 16 12:54:21.408247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.408226 2576 generic.go:356] "Generic (PLEG): container finished" podID="e1f29a63-5601-4894-993e-161c57beae15" containerID="de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b" exitCode=0 Jul 16 12:54:21.408343 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.408262 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" Jul 16 12:54:21.408413 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.408255 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" event={"ID":"e1f29a63-5601-4894-993e-161c57beae15","Type":"ContainerDied","Data":"de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b"} Jul 16 12:54:21.408472 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.408438 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-xbjcn" event={"ID":"e1f29a63-5601-4894-993e-161c57beae15","Type":"ContainerDied","Data":"8af9d1d93080b7edc296ec0b07d8e2a2f23828ebcd043a7554904d30c79f7edf"} Jul 16 12:54:21.414089 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.414075 2576 scope.go:117] "RemoveContainer" containerID="85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba" Jul 16 12:54:21.414363 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:54:21.414341 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba\": container with ID starting with 85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba not found: ID does not exist" containerID="85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba" Jul 16 12:54:21.414406 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.414375 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba"} err="failed to get container status \"85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba\": rpc error: code = NotFound desc = could not find container \"85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba\": container with ID starting with 85c38a40b69168251dae2a786daa88c295f4ab1a4e955cc071efafe869b3adba not found: ID does not exist" Jul 16 12:54:21.414406 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.414390 2576 scope.go:117] "RemoveContainer" containerID="de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b" Jul 16 12:54:21.421402 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.421386 2576 scope.go:117] "RemoveContainer" containerID="de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b" Jul 16 12:54:21.421651 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:54:21.421632 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b\": container with ID starting with de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b not found: ID does not exist" containerID="de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b" Jul 16 12:54:21.421710 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.421657 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b"} err="failed to get container status \"de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b\": rpc error: code = NotFound desc = could not find container \"de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b\": container with ID starting with de6a1ed2d0d8537afca84f885ef29e7d20358cb740403afd6c308a5fc1bb883b not found: ID does not exist" Jul 16 12:54:21.434247 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.434197 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-845fdc676d-t8mnc" podStartSLOduration=1.366810963 podStartE2EDuration="2.43418576s" totalImagesPullingTime="1.067374797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:54:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:54:21.434053828 +0000 UTC m=+546.976660576" watchObservedRunningTime="2026-07-16 12:54:21.43418576 +0000 UTC m=+546.976792506" Jul 16 12:54:21.455469 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.455448 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7fcc644988-gk6dp"] Jul 16 12:54:21.464478 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.464442 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7fcc644988-gk6dp"] Jul 16 12:54:21.477156 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.477130 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-khp4k"] Jul 16 12:54:21.477340 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.477319 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-khp4k" podUID="e17ce232-ae35-452b-985b-e4c19ea3808e" containerName="authorino" containerID="cri-o://d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711" gracePeriod=30 Jul 16 12:54:21.481299 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.481278 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xbjcn"] Jul 16 12:54:21.488083 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.488057 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xbjcn"] Jul 16 12:54:21.737572 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.737547 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-khp4k" Jul 16 12:54:21.803389 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.803358 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk7s8\" (UniqueName: \"kubernetes.io/projected/e17ce232-ae35-452b-985b-e4c19ea3808e-kube-api-access-hk7s8\") pod \"e17ce232-ae35-452b-985b-e4c19ea3808e\" (UID: \"e17ce232-ae35-452b-985b-e4c19ea3808e\") " Jul 16 12:54:21.805325 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.805304 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e17ce232-ae35-452b-985b-e4c19ea3808e-kube-api-access-hk7s8" (OuterVolumeSpecName: "kube-api-access-hk7s8") pod "e17ce232-ae35-452b-985b-e4c19ea3808e" (UID: "e17ce232-ae35-452b-985b-e4c19ea3808e"). InnerVolumeSpecName "kube-api-access-hk7s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:54:21.904880 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:21.904846 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk7s8\" (UniqueName: \"kubernetes.io/projected/e17ce232-ae35-452b-985b-e4c19ea3808e-kube-api-access-hk7s8\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:54:22.414166 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.414127 2576 generic.go:356] "Generic (PLEG): container finished" podID="e17ce232-ae35-452b-985b-e4c19ea3808e" containerID="d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711" exitCode=0 Jul 16 12:54:22.414166 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.414165 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-khp4k" Jul 16 12:54:22.414429 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.414215 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-khp4k" event={"ID":"e17ce232-ae35-452b-985b-e4c19ea3808e","Type":"ContainerDied","Data":"d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711"} Jul 16 12:54:22.414429 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.414264 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-khp4k" event={"ID":"e17ce232-ae35-452b-985b-e4c19ea3808e","Type":"ContainerDied","Data":"4acc38ff908755b49c4fb73e6b4e1412021593667d70c88c25dfa6912805222d"} Jul 16 12:54:22.414429 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.414287 2576 scope.go:117] "RemoveContainer" containerID="d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711" Jul 16 12:54:22.423193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.423166 2576 scope.go:117] "RemoveContainer" containerID="d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711" Jul 16 12:54:22.423527 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:54:22.423504 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711\": container with ID starting with d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711 not found: ID does not exist" containerID="d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711" Jul 16 12:54:22.423628 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.423544 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711"} err="failed to get container status \"d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711\": rpc error: code = NotFound desc = could not find container \"d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711\": container with ID starting with d15091ef12263efc38d3b006f4d17e4b0b7c54bd2f4f820662783ee91c44f711 not found: ID does not exist" Jul 16 12:54:22.458947 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.458928 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-khp4k"] Jul 16 12:54:22.463855 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.463836 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-khp4k"] Jul 16 12:54:22.986416 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.986379 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="501dd6ff-3955-459d-aaa4-16016377db50" path="/var/lib/kubelet/pods/501dd6ff-3955-459d-aaa4-16016377db50/volumes" Jul 16 12:54:22.986913 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.986830 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e17ce232-ae35-452b-985b-e4c19ea3808e" path="/var/lib/kubelet/pods/e17ce232-ae35-452b-985b-e4c19ea3808e/volumes" Jul 16 12:54:22.987185 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:54:22.987166 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f29a63-5601-4894-993e-161c57beae15" path="/var/lib/kubelet/pods/e1f29a63-5601-4894-993e-161c57beae15/volumes" Jul 16 12:55:05.582089 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:05.582059 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:55:14.896932 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:14.896905 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 12:55:14.898998 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:14.898976 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 12:55:14.903422 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:14.903399 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 12:55:14.905429 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:14.905413 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 12:55:26.658534 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:26.658498 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:55:36.211448 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:36.211411 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:55:46.443625 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:46.443582 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:55:55.348685 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:55:55.348651 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:56:06.307934 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.307904 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4"] Jul 16 12:56:06.308299 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308224 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e17ce232-ae35-452b-985b-e4c19ea3808e" containerName="authorino" Jul 16 12:56:06.308299 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308234 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e17ce232-ae35-452b-985b-e4c19ea3808e" containerName="authorino" Jul 16 12:56:06.308299 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308247 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="501dd6ff-3955-459d-aaa4-16016377db50" containerName="authorino" Jul 16 12:56:06.308299 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308254 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="501dd6ff-3955-459d-aaa4-16016377db50" containerName="authorino" Jul 16 12:56:06.308299 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308267 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f29a63-5601-4894-993e-161c57beae15" containerName="authorino" Jul 16 12:56:06.308299 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308272 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f29a63-5601-4894-993e-161c57beae15" containerName="authorino" Jul 16 12:56:06.308492 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308327 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="501dd6ff-3955-459d-aaa4-16016377db50" containerName="authorino" Jul 16 12:56:06.308492 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308336 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f29a63-5601-4894-993e-161c57beae15" containerName="authorino" Jul 16 12:56:06.308492 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.308343 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e17ce232-ae35-452b-985b-e4c19ea3808e" containerName="authorino" Jul 16 12:56:06.311477 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.311458 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.316144 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.316122 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-nnbbp\"" Jul 16 12:56:06.316273 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.316123 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 16 12:56:06.316273 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.316201 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 16 12:56:06.316273 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.316215 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 16 12:56:06.325200 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.325173 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4"] Jul 16 12:56:06.380798 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.380755 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.380978 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.380796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjf8\" (UniqueName: \"kubernetes.io/projected/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-kube-api-access-qqjf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.380978 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.380926 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.380978 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.380966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.381143 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.381061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.381143 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.381095 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482075 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482042 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482114 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjf8\" (UniqueName: \"kubernetes.io/projected/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-kube-api-access-qqjf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482181 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482233 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482478 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482284 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482630 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482605 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482704 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482673 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.482704 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.482681 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.484412 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.484395 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.484668 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.484650 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.493090 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.493064 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjf8\" (UniqueName: \"kubernetes.io/projected/a53b1eb9-e4ab-448d-b41e-2e2572e25aae-kube-api-access-qqjf8\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-2xct4\" (UID: \"a53b1eb9-e4ab-448d-b41e-2e2572e25aae\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.621862 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.621741 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:06.760658 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.760635 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4"] Jul 16 12:56:06.763023 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:56:06.763000 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53b1eb9_e4ab_448d_b41e_2e2572e25aae.slice/crio-04f2f8802ce2d24d5b3e9c58b7ff9e63cd85337d1da1a35db6c374d9bf7903db WatchSource:0}: Error finding container 04f2f8802ce2d24d5b3e9c58b7ff9e63cd85337d1da1a35db6c374d9bf7903db: Status 404 returned error can't find the container with id 04f2f8802ce2d24d5b3e9c58b7ff9e63cd85337d1da1a35db6c374d9bf7903db Jul 16 12:56:06.764363 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.764341 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 12:56:06.880260 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:06.880181 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" event={"ID":"a53b1eb9-e4ab-448d-b41e-2e2572e25aae","Type":"ContainerStarted","Data":"04f2f8802ce2d24d5b3e9c58b7ff9e63cd85337d1da1a35db6c374d9bf7903db"} Jul 16 12:56:08.501419 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:08.501385 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:56:08.889960 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:08.889876 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" event={"ID":"a53b1eb9-e4ab-448d-b41e-2e2572e25aae","Type":"ContainerStarted","Data":"779ea6d285274b988db03fca7181dd89716b170ac5c24ea18c9b16c927a1471d"} Jul 16 12:56:08.933175 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:08.933119 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" podStartSLOduration=1.561285418 podStartE2EDuration="2.933099605s" totalImagesPullingTime="1.371814187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:56:08.932682685 +0000 UTC m=+654.475289433" watchObservedRunningTime="2026-07-16 12:56:08.933099605 +0000 UTC m=+654.475706355" Jul 16 12:56:16.622446 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:16.622406 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:16.622853 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:16.622473 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:16.634806 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:16.634782 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:16.934759 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:16.934732 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-2xct4" Jul 16 12:56:20.315135 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.315104 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj"] Jul 16 12:56:20.318718 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.318698 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.323982 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.323960 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 16 12:56:20.336461 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.336434 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj"] Jul 16 12:56:20.502055 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.502023 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.502055 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.502056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mphg\" (UniqueName: \"kubernetes.io/projected/bc4b2901-1652-4716-a40e-30054c7fcf92-kube-api-access-8mphg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.502253 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.502083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4b2901-1652-4716-a40e-30054c7fcf92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.502253 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.502184 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.502253 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.502247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.502347 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.502272 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.602844 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.602751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.602844 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.602786 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.602844 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.602812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.603092 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.602962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mphg\" (UniqueName: \"kubernetes.io/projected/bc4b2901-1652-4716-a40e-30054c7fcf92-kube-api-access-8mphg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.603092 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.603018 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4b2901-1652-4716-a40e-30054c7fcf92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.603193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.603090 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.603193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.603143 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.603193 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.603184 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.603297 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.603229 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.605267 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.605238 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc4b2901-1652-4716-a40e-30054c7fcf92-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.605484 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.605463 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc4b2901-1652-4716-a40e-30054c7fcf92-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.617367 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.617338 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mphg\" (UniqueName: \"kubernetes.io/projected/bc4b2901-1652-4716-a40e-30054c7fcf92-kube-api-access-8mphg\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj\" (UID: \"bc4b2901-1652-4716-a40e-30054c7fcf92\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.627245 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.627227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:20.763998 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.763969 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj"] Jul 16 12:56:20.767188 ip-10-0-139-160 kubenswrapper[2576]: W0716 12:56:20.767164 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc4b2901_1652_4716_a40e_30054c7fcf92.slice/crio-8ee4e1f8ee1c71b887f335e6e24d4550b12d3a78b0e346e80a58b8d40a6dcb66 WatchSource:0}: Error finding container 8ee4e1f8ee1c71b887f335e6e24d4550b12d3a78b0e346e80a58b8d40a6dcb66: Status 404 returned error can't find the container with id 8ee4e1f8ee1c71b887f335e6e24d4550b12d3a78b0e346e80a58b8d40a6dcb66 Jul 16 12:56:20.948560 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:20.948528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" event={"ID":"bc4b2901-1652-4716-a40e-30054c7fcf92","Type":"ContainerStarted","Data":"8ee4e1f8ee1c71b887f335e6e24d4550b12d3a78b0e346e80a58b8d40a6dcb66"} Jul 16 12:56:21.954175 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:21.954143 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" event={"ID":"bc4b2901-1652-4716-a40e-30054c7fcf92","Type":"ContainerStarted","Data":"97a51f9231a57f4189afbc268d83ff99983c34dfc80b908b557a81e7eee25c3a"} Jul 16 12:56:21.992385 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:21.992338 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" podStartSLOduration=1.726839191 podStartE2EDuration="1.992321347s" totalImagesPullingTime="265.482156ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:56:21.99015173 +0000 UTC m=+667.532758502" watchObservedRunningTime="2026-07-16 12:56:21.992321347 +0000 UTC m=+667.534928072" Jul 16 12:56:22.703519 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:22.703482 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:56:30.628142 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:30.628107 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:30.628517 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:30.628357 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:30.640920 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:30.640893 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:56:31.013342 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:56:31.013314 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj" Jul 16 12:57:54.527906 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.527874 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c8f755c7-wsxk6"] Jul 16 12:57:54.531444 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.531421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c8f755c7-wsxk6" Jul 16 12:57:54.542494 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.542469 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c8f755c7-wsxk6"] Jul 16 12:57:54.625993 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.625958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwkts\" (UniqueName: \"kubernetes.io/projected/f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a-kube-api-access-lwkts\") pod \"authorino-56c8f755c7-wsxk6\" (UID: \"f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a\") " pod="kuadrant-system/authorino-56c8f755c7-wsxk6" Jul 16 12:57:54.626137 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.626039 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a-tls-cert\") pod \"authorino-56c8f755c7-wsxk6\" (UID: \"f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a\") " pod="kuadrant-system/authorino-56c8f755c7-wsxk6" Jul 16 12:57:54.727459 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.727426 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a-tls-cert\") pod \"authorino-56c8f755c7-wsxk6\" (UID: \"f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a\") " pod="kuadrant-system/authorino-56c8f755c7-wsxk6" Jul 16 12:57:54.727790 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.727487 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwkts\" (UniqueName: \"kubernetes.io/projected/f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a-kube-api-access-lwkts\") pod \"authorino-56c8f755c7-wsxk6\" (UID: \"f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a\") " pod="kuadrant-system/authorino-56c8f755c7-wsxk6" Jul 16 12:57:54.730080 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.730055 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a-tls-cert\") pod \"authorino-56c8f755c7-wsxk6\" (UID: \"f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a\") " pod="kuadrant-system/authorino-56c8f755c7-wsxk6" Jul 16 12:57:54.742191 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.742160 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwkts\" (UniqueName: \"kubernetes.io/projected/f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a-kube-api-access-lwkts\") pod \"authorino-56c8f755c7-wsxk6\" (UID: \"f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a\") " pod="kuadrant-system/authorino-56c8f755c7-wsxk6" Jul 16 12:57:54.840495 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.840417 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c8f755c7-wsxk6" Jul 16 12:57:54.995386 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:54.995341 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c8f755c7-wsxk6"] Jul 16 12:57:55.385653 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:55.385605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c8f755c7-wsxk6" event={"ID":"f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a","Type":"ContainerStarted","Data":"5ab113e6aff0274977030911b58b682390c80010665d5dc497de76e56bbd9fad"} Jul 16 12:57:56.393684 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.393645 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c8f755c7-wsxk6" event={"ID":"f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a","Type":"ContainerStarted","Data":"a774bcc5348b79800cf9f099758b923126567939d22bd203795daa0e3b7a5dfe"} Jul 16 12:57:56.428699 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.428619 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-56c8f755c7-wsxk6" podStartSLOduration=2.017895881 podStartE2EDuration="2.428579431s" totalImagesPullingTime="410.68355ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 12:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 12:57:56.425655758 +0000 UTC m=+761.968262505" watchObservedRunningTime="2026-07-16 12:57:56.428579431 +0000 UTC m=+761.971186179" Jul 16 12:57:56.470493 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.470456 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-845fdc676d-t8mnc"] Jul 16 12:57:56.470684 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.470658 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-845fdc676d-t8mnc" podUID="59da6f90-846a-4ce9-8694-6aa441962a31" containerName="authorino" containerID="cri-o://a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe" gracePeriod=30 Jul 16 12:57:56.727328 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.727295 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:57:56.850061 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.850031 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59da6f90-846a-4ce9-8694-6aa441962a31-tls-cert\") pod \"59da6f90-846a-4ce9-8694-6aa441962a31\" (UID: \"59da6f90-846a-4ce9-8694-6aa441962a31\") " Jul 16 12:57:56.850229 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.850095 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qhr9\" (UniqueName: \"kubernetes.io/projected/59da6f90-846a-4ce9-8694-6aa441962a31-kube-api-access-9qhr9\") pod \"59da6f90-846a-4ce9-8694-6aa441962a31\" (UID: \"59da6f90-846a-4ce9-8694-6aa441962a31\") " Jul 16 12:57:56.852131 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.852091 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59da6f90-846a-4ce9-8694-6aa441962a31-kube-api-access-9qhr9" (OuterVolumeSpecName: "kube-api-access-9qhr9") pod "59da6f90-846a-4ce9-8694-6aa441962a31" (UID: "59da6f90-846a-4ce9-8694-6aa441962a31"). InnerVolumeSpecName "kube-api-access-9qhr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 12:57:56.859656 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.859628 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59da6f90-846a-4ce9-8694-6aa441962a31-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "59da6f90-846a-4ce9-8694-6aa441962a31" (UID: "59da6f90-846a-4ce9-8694-6aa441962a31"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 12:57:56.951786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.951751 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/59da6f90-846a-4ce9-8694-6aa441962a31-tls-cert\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:57:56.951786 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:56.951780 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qhr9\" (UniqueName: \"kubernetes.io/projected/59da6f90-846a-4ce9-8694-6aa441962a31-kube-api-access-9qhr9\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 12:57:57.398975 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.398895 2576 generic.go:356] "Generic (PLEG): container finished" podID="59da6f90-846a-4ce9-8694-6aa441962a31" containerID="a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe" exitCode=0 Jul 16 12:57:57.398975 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.398935 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845fdc676d-t8mnc" Jul 16 12:57:57.399450 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.398985 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845fdc676d-t8mnc" event={"ID":"59da6f90-846a-4ce9-8694-6aa441962a31","Type":"ContainerDied","Data":"a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe"} Jul 16 12:57:57.399450 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.399034 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845fdc676d-t8mnc" event={"ID":"59da6f90-846a-4ce9-8694-6aa441962a31","Type":"ContainerDied","Data":"11147faed370608658b66a75ee96da7a5dcfb44cf74a6c499589656950b76944"} Jul 16 12:57:57.399450 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.399058 2576 scope.go:117] "RemoveContainer" containerID="a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe" Jul 16 12:57:57.407011 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.406995 2576 scope.go:117] "RemoveContainer" containerID="a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe" Jul 16 12:57:57.407243 ip-10-0-139-160 kubenswrapper[2576]: E0716 12:57:57.407226 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe\": container with ID starting with a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe not found: ID does not exist" containerID="a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe" Jul 16 12:57:57.407285 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.407252 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe"} err="failed to get container status \"a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe\": rpc error: code = NotFound desc = could not find container \"a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe\": container with ID starting with a14349b27c7584b7e37b5e352e74a09d76aa668cffdc25bbff799a3ebebe5ffe not found: ID does not exist" Jul 16 12:57:57.424266 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.424247 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-845fdc676d-t8mnc"] Jul 16 12:57:57.430100 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:57.430082 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-845fdc676d-t8mnc"] Jul 16 12:57:58.985943 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:57:58.985908 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59da6f90-846a-4ce9-8694-6aa441962a31" path="/var/lib/kubelet/pods/59da6f90-846a-4ce9-8694-6aa441962a31/volumes" Jul 16 12:58:26.615387 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:58:26.615351 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:58:36.395849 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:58:36.395756 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:58:47.319442 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:58:47.319411 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:58:56.319611 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:58:56.319580 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:59:07.595749 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:59:07.595715 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:59:14.961179 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:59:14.961154 2576 scope.go:117] "RemoveContainer" containerID="d4b8dc8c5c250e82f4add216acdee2d84991cfb17dd898bb33e2dd9453e5b0a7" Jul 16 12:59:35.700876 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:59:35.700843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 12:59:47.302359 ip-10-0-139-160 kubenswrapper[2576]: I0716 12:59:47.302322 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:00:14.929673 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:00:14.929597 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:00:14.929673 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:00:14.929633 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:00:14.936479 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:00:14.936454 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:00:14.936596 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:00:14.936460 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:00:27.131532 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:00:27.131501 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:00:40.210210 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:00:40.210176 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:00:48.705462 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:00:48.705428 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:00:56.209438 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:00:56.209405 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:01:07.233714 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:01:07.233682 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:01:17.617425 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:01:17.617377 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:01:28.796656 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:01:28.796624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:01:38.996448 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:01:38.996360 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:02:07.304160 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:02:07.304125 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:02:19.655709 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:02:19.655672 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:02:43.999923 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:02:43.999882 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:02:54.592482 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:02:54.592446 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:03:10.398038 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:03:10.397953 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:03:21.600680 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:03:21.600642 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:03:31.246406 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:03:31.246369 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:03:41.294509 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:03:41.294474 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:03:50.996965 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:03:50.996926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:05:14.957561 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:05:14.957532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:05:14.958551 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:05:14.958532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:05:14.963689 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:05:14.963672 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:05:14.964498 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:05:14.964481 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:05:40.416582 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:05:40.416476 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:05:49.908606 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:05:49.908570 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:06:02.515113 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:06:02.515073 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:06:15.806430 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:06:15.806335 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:06:25.798147 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:06:25.798111 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:06:35.697150 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:06:35.697117 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:06:45.921648 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:06:45.921615 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:06:56.302888 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:06:56.302852 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:07:06.940656 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:07:06.940624 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:07:17.004948 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:07:17.004913 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:07:30.314700 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:07:30.314661 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:07:54.116312 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:07:54.116279 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:08:01.707977 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:08:01.707944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:08:11.800838 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:08:11.800789 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:08:22.319615 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:08:22.319578 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:08:33.375618 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:08:33.375587 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:08:45.809264 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:08:45.809230 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:08:57.199012 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:08:57.198975 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:09:07.305504 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:09:07.305421 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:09:15.121923 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:09:15.121895 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:09:26.606848 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:09:26.606160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:09:36.727866 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:09:36.727834 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:09:49.915259 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:09:49.915224 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:10:04.699194 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:04.699158 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:10:14.330713 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:14.330676 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:10:14.984765 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:14.984733 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:10:14.986910 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:14.986892 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:10:14.991045 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:14.991026 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:10:14.992988 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:14.992973 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:10:25.199417 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:25.199386 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:10:36.397677 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:36.397590 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:10:56.108973 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:10:56.108932 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:11:07.726737 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:11:07.726698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:11:20.306908 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:11:20.306875 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:11:30.105514 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:11:30.105485 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:11:44.044663 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:11:44.044633 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:11:58.628907 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:11:58.628876 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:12:09.605919 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:12:09.605842 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:13:19.911181 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:13:19.911149 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:13:39.190607 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:13:39.190572 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:14:41.588284 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:14:41.588249 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:14:54.718152 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:14:54.718116 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:15:15.011336 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:15:15.011308 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:15:15.014271 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:15:15.014241 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:15:15.017618 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:15:15.017602 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:15:15.021145 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:15:15.021126 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:16:06.389344 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:16:06.389308 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:16:28.591614 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:16:28.591576 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:17:38.887305 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:17:38.887271 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:19:16.799909 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:19:16.799873 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:19:25.990049 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:19:25.990012 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:19:41.391260 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:19:41.391222 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:19:59.986109 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:19:59.986074 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:20:15.038881 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:20:15.038852 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:20:15.050493 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:20:15.050454 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:20:15.052123 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:20:15.052100 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:20:15.059658 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:20:15.059637 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:20:40.894247 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:20:40.894214 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:20:52.705372 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:20:52.705337 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:21:17.599854 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:21:17.599806 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:21:38.288521 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:21:38.288486 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:22:55.584255 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:22:55.584221 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:23:07.188357 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:23:07.188323 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:24:01.083085 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:01.083049 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:24:15.585111 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:15.585030 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:24:20.913831 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.913780 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc"] Jul 16 13:24:20.914209 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.914133 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59da6f90-846a-4ce9-8694-6aa441962a31" containerName="authorino" Jul 16 13:24:20.914209 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.914144 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="59da6f90-846a-4ce9-8694-6aa441962a31" containerName="authorino" Jul 16 13:24:20.914209 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.914205 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="59da6f90-846a-4ce9-8694-6aa441962a31" containerName="authorino" Jul 16 13:24:20.917074 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.917054 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:20.920461 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.920444 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-00668a\"/\"test-model-00668a-kserve-self-signed-certs\"" Jul 16 13:24:20.921448 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.921431 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-00668a\"/\"kube-root-ca.crt\"" Jul 16 13:24:20.921986 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.921971 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-00668a\"/\"openshift-service-ca.crt\"" Jul 16 13:24:20.922200 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.922183 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-00668a\"/\"default-dockercfg-nnvts\"" Jul 16 13:24:20.931553 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.931530 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc"] Jul 16 13:24:20.996358 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.996323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4hv2\" (UniqueName: \"kubernetes.io/projected/b71c2bfb-1ae4-47e7-bff5-193d6961504e-kube-api-access-p4hv2\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:20.996518 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.996394 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tmp-dir\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:20.996518 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.996425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tls-certs\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:20.996518 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.996475 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-dshm\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:20.996518 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.996506 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-home\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:20.996716 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:20.996534 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-model-cache\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.097896 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.097866 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tmp-dir\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.097896 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.097900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tls-certs\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.098142 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.098088 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-dshm\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.098142 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.098109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-home\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.098142 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.098129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-model-cache\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.098318 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.098187 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4hv2\" (UniqueName: \"kubernetes.io/projected/b71c2bfb-1ae4-47e7-bff5-193d6961504e-kube-api-access-p4hv2\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.098318 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.098294 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tmp-dir\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.098486 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.098458 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-model-cache\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.098554 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.098501 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-home\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.100316 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.100296 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-dshm\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.100470 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.100454 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tls-certs\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.108622 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.108600 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4hv2\" (UniqueName: \"kubernetes.io/projected/b71c2bfb-1ae4-47e7-bff5-193d6961504e-kube-api-access-p4hv2\") pod \"test-model-00668a-kserve-75d74cf5cf-v5wtc\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.226264 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.226233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:21.359729 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.359706 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc"] Jul 16 13:24:21.361779 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:24:21.361747 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb71c2bfb_1ae4_47e7_bff5_193d6961504e.slice/crio-7f3754374c3de45e010995a065ad7f7393ec00c21fe9e5ed49d14f6124b8dac5 WatchSource:0}: Error finding container 7f3754374c3de45e010995a065ad7f7393ec00c21fe9e5ed49d14f6124b8dac5: Status 404 returned error can't find the container with id 7f3754374c3de45e010995a065ad7f7393ec00c21fe9e5ed49d14f6124b8dac5 Jul 16 13:24:21.439643 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.439605 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" event={"ID":"b71c2bfb-1ae4-47e7-bff5-193d6961504e","Type":"ContainerStarted","Data":"59e0d4d5f5932a3798e48717745ab76419b59900d731613746fda6df943440fe"} Jul 16 13:24:21.439801 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.439650 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" event={"ID":"b71c2bfb-1ae4-47e7-bff5-193d6961504e","Type":"ContainerStarted","Data":"7f3754374c3de45e010995a065ad7f7393ec00c21fe9e5ed49d14f6124b8dac5"} Jul 16 13:24:21.459177 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:21.459122 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" podStartSLOduration=1.4591012239999999 podStartE2EDuration="1.459101224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:24:21.458569062 +0000 UTC m=+2347.001175809" watchObservedRunningTime="2026-07-16 13:24:21.459101224 +0000 UTC m=+2347.001707973" Jul 16 13:24:25.185983 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:25.185951 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:24:31.226931 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:31.226894 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:31.226931 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:31.226938 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:31.240346 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:31.240320 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:31.498351 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:31.498269 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:24:36.293710 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:36.293674 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:24:52.887537 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:24:52.887500 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:25:08.466574 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:08.466543 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc"] Jul 16 13:25:13.290971 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:13.290934 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:25:15.076375 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:15.076346 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:25:15.088476 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:15.088448 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:25:15.090099 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:15.090077 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:25:15.095318 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:15.095300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:25:23.486693 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.486631 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" podUID="b71c2bfb-1ae4-47e7-bff5-193d6961504e" containerName="main" containerID="cri-o://59e0d4d5f5932a3798e48717745ab76419b59900d731613746fda6df943440fe" gracePeriod=45 Jul 16 13:25:23.717307 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.717272 2576 generic.go:356] "Generic (PLEG): container finished" podID="b71c2bfb-1ae4-47e7-bff5-193d6961504e" containerID="59e0d4d5f5932a3798e48717745ab76419b59900d731613746fda6df943440fe" exitCode=0 Jul 16 13:25:23.717457 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.717356 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" event={"ID":"b71c2bfb-1ae4-47e7-bff5-193d6961504e","Type":"ContainerDied","Data":"59e0d4d5f5932a3798e48717745ab76419b59900d731613746fda6df943440fe"} Jul 16 13:25:23.736271 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.736243 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:25:23.762385 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762304 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tmp-dir\") pod \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " Jul 16 13:25:23.762385 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762348 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4hv2\" (UniqueName: \"kubernetes.io/projected/b71c2bfb-1ae4-47e7-bff5-193d6961504e-kube-api-access-p4hv2\") pod \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " Jul 16 13:25:23.762606 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762395 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tls-certs\") pod \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " Jul 16 13:25:23.762606 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762435 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-home\") pod \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " Jul 16 13:25:23.762606 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762464 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-dshm\") pod \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " Jul 16 13:25:23.762606 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762583 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-model-cache\") pod \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\" (UID: \"b71c2bfb-1ae4-47e7-bff5-193d6961504e\") " Jul 16 13:25:23.762606 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762589 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b71c2bfb-1ae4-47e7-bff5-193d6961504e" (UID: "b71c2bfb-1ae4-47e7-bff5-193d6961504e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:25:23.762900 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762853 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-home" (OuterVolumeSpecName: "home") pod "b71c2bfb-1ae4-47e7-bff5-193d6961504e" (UID: "b71c2bfb-1ae4-47e7-bff5-193d6961504e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:25:23.762973 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762920 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b71c2bfb-1ae4-47e7-bff5-193d6961504e" (UID: "b71c2bfb-1ae4-47e7-bff5-193d6961504e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:25:23.763025 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762963 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tmp-dir\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:25:23.763025 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.762990 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-home\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:25:23.764838 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.764782 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b71c2bfb-1ae4-47e7-bff5-193d6961504e" (UID: "b71c2bfb-1ae4-47e7-bff5-193d6961504e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:25:23.764944 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.764843 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-dshm" (OuterVolumeSpecName: "dshm") pod "b71c2bfb-1ae4-47e7-bff5-193d6961504e" (UID: "b71c2bfb-1ae4-47e7-bff5-193d6961504e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:25:23.764944 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.764853 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71c2bfb-1ae4-47e7-bff5-193d6961504e-kube-api-access-p4hv2" (OuterVolumeSpecName: "kube-api-access-p4hv2") pod "b71c2bfb-1ae4-47e7-bff5-193d6961504e" (UID: "b71c2bfb-1ae4-47e7-bff5-193d6961504e"). InnerVolumeSpecName "kube-api-access-p4hv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:25:23.863601 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.863566 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4hv2\" (UniqueName: \"kubernetes.io/projected/b71c2bfb-1ae4-47e7-bff5-193d6961504e-kube-api-access-p4hv2\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:25:23.863601 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.863597 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b71c2bfb-1ae4-47e7-bff5-193d6961504e-tls-certs\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:25:23.863601 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.863607 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-dshm\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:25:23.863851 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:23.863616 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b71c2bfb-1ae4-47e7-bff5-193d6961504e-model-cache\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:25:24.723628 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:24.723592 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" Jul 16 13:25:24.723628 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:24.723622 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc" event={"ID":"b71c2bfb-1ae4-47e7-bff5-193d6961504e","Type":"ContainerDied","Data":"7f3754374c3de45e010995a065ad7f7393ec00c21fe9e5ed49d14f6124b8dac5"} Jul 16 13:25:24.724211 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:24.723673 2576 scope.go:117] "RemoveContainer" containerID="59e0d4d5f5932a3798e48717745ab76419b59900d731613746fda6df943440fe" Jul 16 13:25:24.747330 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:24.747296 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc"] Jul 16 13:25:24.750146 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:24.750120 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-00668a/test-model-00668a-kserve-75d74cf5cf-v5wtc"] Jul 16 13:25:24.789645 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:24.789608 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:25:24.986243 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:24.986149 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b71c2bfb-1ae4-47e7-bff5-193d6961504e" path="/var/lib/kubelet/pods/b71c2bfb-1ae4-47e7-bff5-193d6961504e/volumes" Jul 16 13:25:49.792960 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:25:49.792918 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:26:11.192508 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:26:11.192473 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:26:21.389161 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:26:21.389126 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:26:40.790909 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:26:40.790874 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:27:06.385151 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:06.385072 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:27:27.690793 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:27.690756 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:27:43.186286 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:43.186254 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:27:45.685422 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.685390 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr"] Jul 16 13:27:45.685780 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.685728 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b71c2bfb-1ae4-47e7-bff5-193d6961504e" containerName="main" Jul 16 13:27:45.685780 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.685739 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b71c2bfb-1ae4-47e7-bff5-193d6961504e" containerName="main" Jul 16 13:27:45.685876 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.685795 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b71c2bfb-1ae4-47e7-bff5-193d6961504e" containerName="main" Jul 16 13:27:45.688864 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.688849 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.692020 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.691999 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"kube-root-ca.crt\"" Jul 16 13:27:45.693534 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.693516 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"default-dockercfg-zlwnw\"" Jul 16 13:27:45.693642 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.693534 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"auth-test-model-978d92-kserve-self-signed-certs\"" Jul 16 13:27:45.693642 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.693549 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"openshift-service-ca.crt\"" Jul 16 13:27:45.700121 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.700101 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr"] Jul 16 13:27:45.806537 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.806499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-model-cache\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.806537 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.806535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-tmp-dir\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.806762 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.806597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-home\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.806762 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.806693 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-dshm\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.806762 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.806712 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22215952-cf02-4beb-9635-b129171357e4-tls-certs\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.806762 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.806740 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwk9\" (UniqueName: \"kubernetes.io/projected/22215952-cf02-4beb-9635-b129171357e4-kube-api-access-vzwk9\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908049 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-model-cache\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908049 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-tmp-dir\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908275 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908078 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-home\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908275 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-dshm\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908275 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22215952-cf02-4beb-9635-b129171357e4-tls-certs\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908275 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908214 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwk9\" (UniqueName: \"kubernetes.io/projected/22215952-cf02-4beb-9635-b129171357e4-kube-api-access-vzwk9\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908471 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908436 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-model-cache\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908523 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908499 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-tmp-dir\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.908580 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.908503 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-home\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.910422 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.910397 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-dshm\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.910689 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.910668 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22215952-cf02-4beb-9635-b129171357e4-tls-certs\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.919399 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.919380 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwk9\" (UniqueName: \"kubernetes.io/projected/22215952-cf02-4beb-9635-b129171357e4-kube-api-access-vzwk9\") pod \"auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:45.998246 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:45.998157 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:46.129241 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:46.129212 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr"] Jul 16 13:27:46.131176 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:27:46.131151 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22215952_cf02_4beb_9635_b129171357e4.slice/crio-0d8e1a05e5b22b22ed19f566c696f3da4911e09310379d5b1429dcab46d546d3 WatchSource:0}: Error finding container 0d8e1a05e5b22b22ed19f566c696f3da4911e09310379d5b1429dcab46d546d3: Status 404 returned error can't find the container with id 0d8e1a05e5b22b22ed19f566c696f3da4911e09310379d5b1429dcab46d546d3 Jul 16 13:27:46.359671 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:46.359586 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" event={"ID":"22215952-cf02-4beb-9635-b129171357e4","Type":"ContainerStarted","Data":"9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190"} Jul 16 13:27:46.359671 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:46.359623 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" event={"ID":"22215952-cf02-4beb-9635-b129171357e4","Type":"ContainerStarted","Data":"0d8e1a05e5b22b22ed19f566c696f3da4911e09310379d5b1429dcab46d546d3"} Jul 16 13:27:46.382973 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:46.382928 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" podStartSLOduration=1.382913652 podStartE2EDuration="1.382913652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:27:46.381987307 +0000 UTC m=+2551.924594055" watchObservedRunningTime="2026-07-16 13:27:46.382913652 +0000 UTC m=+2551.925520415" Jul 16 13:27:55.999356 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:55.999303 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:55.999356 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:55.999374 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:56.011723 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:56.011701 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:56.416130 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:56.416047 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:27:57.207124 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:27:57.207092 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:28:15.989158 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:28:15.989124 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:28:35.987168 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:28:35.987076 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:28:53.890102 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:28:53.890068 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:29:12.484691 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:29:12.484659 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:29:30.802387 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:29:30.802350 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:29:50.227949 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:29:50.227908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:30:07.309806 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:07.309725 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:30:15.110028 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:15.110003 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:30:15.116061 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:15.116041 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:30:15.117171 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:15.117154 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:30:15.122679 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:15.122664 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:30:21.391735 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:21.391701 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr"] Jul 16 13:30:24.898222 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:24.898181 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:30:36.411495 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.411448 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" podUID="22215952-cf02-4beb-9635-b129171357e4" containerName="main" containerID="cri-o://9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190" gracePeriod=45 Jul 16 13:30:36.415162 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.415125 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" podUID="22215952-cf02-4beb-9635-b129171357e4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": EOF" Jul 16 13:30:36.698887 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.698860 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:30:36.717324 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.717300 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzwk9\" (UniqueName: \"kubernetes.io/projected/22215952-cf02-4beb-9635-b129171357e4-kube-api-access-vzwk9\") pod \"22215952-cf02-4beb-9635-b129171357e4\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " Jul 16 13:30:36.717415 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.717332 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-dshm\") pod \"22215952-cf02-4beb-9635-b129171357e4\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " Jul 16 13:30:36.717415 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.717388 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-tmp-dir\") pod \"22215952-cf02-4beb-9635-b129171357e4\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " Jul 16 13:30:36.717496 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.717415 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-model-cache\") pod \"22215952-cf02-4beb-9635-b129171357e4\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " Jul 16 13:30:36.717496 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.717439 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-home\") pod \"22215952-cf02-4beb-9635-b129171357e4\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " Jul 16 13:30:36.717672 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.717643 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "22215952-cf02-4beb-9635-b129171357e4" (UID: "22215952-cf02-4beb-9635-b129171357e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:30:36.717736 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.717656 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "22215952-cf02-4beb-9635-b129171357e4" (UID: "22215952-cf02-4beb-9635-b129171357e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:30:36.717780 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.717750 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-home" (OuterVolumeSpecName: "home") pod "22215952-cf02-4beb-9635-b129171357e4" (UID: "22215952-cf02-4beb-9635-b129171357e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:30:36.719420 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.719398 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-dshm" (OuterVolumeSpecName: "dshm") pod "22215952-cf02-4beb-9635-b129171357e4" (UID: "22215952-cf02-4beb-9635-b129171357e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:30:36.719511 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.719461 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22215952-cf02-4beb-9635-b129171357e4-kube-api-access-vzwk9" (OuterVolumeSpecName: "kube-api-access-vzwk9") pod "22215952-cf02-4beb-9635-b129171357e4" (UID: "22215952-cf02-4beb-9635-b129171357e4"). InnerVolumeSpecName "kube-api-access-vzwk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:30:36.818055 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.818021 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22215952-cf02-4beb-9635-b129171357e4-tls-certs\") pod \"22215952-cf02-4beb-9635-b129171357e4\" (UID: \"22215952-cf02-4beb-9635-b129171357e4\") " Jul 16 13:30:36.818258 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.818246 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwk9\" (UniqueName: \"kubernetes.io/projected/22215952-cf02-4beb-9635-b129171357e4-kube-api-access-vzwk9\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:30:36.818303 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.818260 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-dshm\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:30:36.818303 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.818269 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-tmp-dir\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:30:36.818303 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.818279 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-model-cache\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:30:36.818303 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.818287 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22215952-cf02-4beb-9635-b129171357e4-home\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:30:36.820062 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.820039 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22215952-cf02-4beb-9635-b129171357e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "22215952-cf02-4beb-9635-b129171357e4" (UID: "22215952-cf02-4beb-9635-b129171357e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:30:36.925094 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:36.919374 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22215952-cf02-4beb-9635-b129171357e4-tls-certs\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:30:37.123779 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.123742 2576 generic.go:356] "Generic (PLEG): container finished" podID="22215952-cf02-4beb-9635-b129171357e4" containerID="9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190" exitCode=0 Jul 16 13:30:37.123956 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.123832 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" Jul 16 13:30:37.123956 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.123845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" event={"ID":"22215952-cf02-4beb-9635-b129171357e4","Type":"ContainerDied","Data":"9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190"} Jul 16 13:30:37.123956 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.123885 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr" event={"ID":"22215952-cf02-4beb-9635-b129171357e4","Type":"ContainerDied","Data":"0d8e1a05e5b22b22ed19f566c696f3da4911e09310379d5b1429dcab46d546d3"} Jul 16 13:30:37.123956 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.123901 2576 scope.go:117] "RemoveContainer" containerID="9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190" Jul 16 13:30:37.135469 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.135434 2576 scope.go:117] "RemoveContainer" containerID="9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190" Jul 16 13:30:37.135747 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:30:37.135722 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190\": container with ID starting with 9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190 not found: ID does not exist" containerID="9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190" Jul 16 13:30:37.135856 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.135753 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190"} err="failed to get container status \"9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190\": rpc error: code = NotFound desc = could not find container \"9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190\": container with ID starting with 9a308a2b3eb28cb6f81ee645716b6ee7d8ded0f6e1845510b5200b90858f3190 not found: ID does not exist" Jul 16 13:30:37.146585 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.146559 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr"] Jul 16 13:30:37.155743 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:37.155710 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/auth-test-model-978d92-kserve-5dfccdc9ff-s4fcr"] Jul 16 13:30:38.986717 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:38.986686 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22215952-cf02-4beb-9635-b129171357e4" path="/var/lib/kubelet/pods/22215952-cf02-4beb-9635-b129171357e4/volumes" Jul 16 13:30:43.295948 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:43.295913 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:30:46.474128 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.474086 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99"] Jul 16 13:30:46.474483 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.474448 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22215952-cf02-4beb-9635-b129171357e4" containerName="main" Jul 16 13:30:46.474483 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.474460 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22215952-cf02-4beb-9635-b129171357e4" containerName="main" Jul 16 13:30:46.474560 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.474545 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22215952-cf02-4beb-9635-b129171357e4" containerName="main" Jul 16 13:30:46.478747 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.478727 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.481594 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.481574 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"kube-root-ca.crt\"" Jul 16 13:30:46.483030 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.483012 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"sub-test-model-c7dd05-kserve-self-signed-certs\"" Jul 16 13:30:46.483128 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.483055 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"openshift-service-ca.crt\"" Jul 16 13:30:46.483128 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.483056 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-c7dd05\"/\"default-dockercfg-hpcfq\"" Jul 16 13:30:46.489476 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.489456 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99"] Jul 16 13:30:46.504526 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.504503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tmp-dir\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.504613 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.504547 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tls-certs\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.504613 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.504573 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpfpd\" (UniqueName: \"kubernetes.io/projected/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-kube-api-access-xpfpd\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.504613 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.504606 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-home\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.504732 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.504624 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-model-cache\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.504732 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.504702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-dshm\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605191 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tmp-dir\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605350 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605204 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tls-certs\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605350 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605322 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpfpd\" (UniqueName: \"kubernetes.io/projected/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-kube-api-access-xpfpd\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605431 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-home\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605431 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605414 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-model-cache\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605500 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605444 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-dshm\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605563 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605544 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tmp-dir\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605698 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605680 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-home\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.605759 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.605739 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-model-cache\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.607630 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.607610 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-dshm\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.607752 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.607728 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tls-certs\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.616033 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.616003 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpfpd\" (UniqueName: \"kubernetes.io/projected/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-kube-api-access-xpfpd\") pod \"sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.788124 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.788042 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:46.918027 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:46.917999 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99"] Jul 16 13:30:46.920093 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:30:46.920062 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb6bdf50_a414_4bf1_b21d_c94c5180ba24.slice/crio-a63bc46da0500ed8801628af042a6e55b1c29f63f4fca7a3e4b0650ca79cca96 WatchSource:0}: Error finding container a63bc46da0500ed8801628af042a6e55b1c29f63f4fca7a3e4b0650ca79cca96: Status 404 returned error can't find the container with id a63bc46da0500ed8801628af042a6e55b1c29f63f4fca7a3e4b0650ca79cca96 Jul 16 13:30:47.172479 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:47.172445 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" event={"ID":"bb6bdf50-a414-4bf1-b21d-c94c5180ba24","Type":"ContainerStarted","Data":"722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185"} Jul 16 13:30:47.172479 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:47.172484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" event={"ID":"bb6bdf50-a414-4bf1-b21d-c94c5180ba24","Type":"ContainerStarted","Data":"a63bc46da0500ed8801628af042a6e55b1c29f63f4fca7a3e4b0650ca79cca96"} Jul 16 13:30:56.788645 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:56.788604 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:56.789059 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:56.788660 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:56.801268 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:56.801244 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:30:56.831046 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:56.831004 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" podStartSLOduration=10.830987217 podStartE2EDuration="10.830987217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:30:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:30:47.192582354 +0000 UTC m=+2732.735189101" watchObservedRunningTime="2026-07-16 13:30:56.830987217 +0000 UTC m=+2742.373593978" Jul 16 13:30:57.230624 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:30:57.230596 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:31:00.901709 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:00.901677 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:31:29.401947 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:29.401912 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:31:35.914871 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:35.914779 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99"] Jul 16 13:31:44.398303 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.398275 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7"] Jul 16 13:31:44.401656 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.401641 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.404673 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.404652 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"kube-root-ca.crt\"" Jul 16 13:31:44.406074 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.406057 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"rate-test-model-978d92-kserve-self-signed-certs\"" Jul 16 13:31:44.406179 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.406097 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"openshift-service-ca.crt\"" Jul 16 13:31:44.406179 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.406057 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-978d92\"/\"default-dockercfg-zlwnw\"" Jul 16 13:31:44.411010 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.410987 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7"] Jul 16 13:31:44.522757 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.522726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-tmp-dir\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.522963 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.522764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-model-cache\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.522963 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.522796 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-dshm\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.522963 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.522891 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-home\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.522963 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.522922 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2898e66d-598b-4289-814c-4b54f109ea68-tls-certs\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.523141 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.522964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6clk\" (UniqueName: \"kubernetes.io/projected/2898e66d-598b-4289-814c-4b54f109ea68-kube-api-access-g6clk\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624216 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624182 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-tmp-dir\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624216 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-model-cache\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624444 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-dshm\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624444 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624302 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-home\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624444 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2898e66d-598b-4289-814c-4b54f109ea68-tls-certs\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624444 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6clk\" (UniqueName: \"kubernetes.io/projected/2898e66d-598b-4289-814c-4b54f109ea68-kube-api-access-g6clk\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624678 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624656 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-tmp-dir\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624755 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624696 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-home\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.624755 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.624705 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-model-cache\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.626694 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.626671 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-dshm\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.626846 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.626812 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2898e66d-598b-4289-814c-4b54f109ea68-tls-certs\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.636362 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.636342 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6clk\" (UniqueName: \"kubernetes.io/projected/2898e66d-598b-4289-814c-4b54f109ea68-kube-api-access-g6clk\") pod \"rate-test-model-978d92-kserve-856f88c967-276g7\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.711418 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.711381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:44.841541 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:44.841516 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7"] Jul 16 13:31:44.843397 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:31:44.843367 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2898e66d_598b_4289_814c_4b54f109ea68.slice/crio-44a5067b69729d192dd75688147eb2bc5a508960a5012027800b790ce54f59ae WatchSource:0}: Error finding container 44a5067b69729d192dd75688147eb2bc5a508960a5012027800b790ce54f59ae: Status 404 returned error can't find the container with id 44a5067b69729d192dd75688147eb2bc5a508960a5012027800b790ce54f59ae Jul 16 13:31:45.435567 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:45.435530 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" event={"ID":"2898e66d-598b-4289-814c-4b54f109ea68","Type":"ContainerStarted","Data":"a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc"} Jul 16 13:31:45.436024 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:45.435580 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" event={"ID":"2898e66d-598b-4289-814c-4b54f109ea68","Type":"ContainerStarted","Data":"44a5067b69729d192dd75688147eb2bc5a508960a5012027800b790ce54f59ae"} Jul 16 13:31:45.463317 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:45.463266 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" podStartSLOduration=1.46325153 podStartE2EDuration="1.46325153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 13:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:31:45.462926546 +0000 UTC m=+2791.005533295" watchObservedRunningTime="2026-07-16 13:31:45.46325153 +0000 UTC m=+2791.005858276" Jul 16 13:31:49.498397 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:49.498358 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:31:50.934586 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:50.934538 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" podUID="bb6bdf50-a414-4bf1-b21d-c94c5180ba24" containerName="main" containerID="cri-o://722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185" gracePeriod=45 Jul 16 13:31:51.183858 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.183830 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:31:51.286129 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286047 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tmp-dir\") pod \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " Jul 16 13:31:51.286129 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286080 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-dshm\") pod \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " Jul 16 13:31:51.286332 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286146 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tls-certs\") pod \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " Jul 16 13:31:51.286332 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286165 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-model-cache\") pod \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " Jul 16 13:31:51.286332 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286191 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpfpd\" (UniqueName: \"kubernetes.io/projected/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-kube-api-access-xpfpd\") pod \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " Jul 16 13:31:51.286332 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286241 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-home\") pod \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\" (UID: \"bb6bdf50-a414-4bf1-b21d-c94c5180ba24\") " Jul 16 13:31:51.286332 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286302 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb6bdf50-a414-4bf1-b21d-c94c5180ba24" (UID: "bb6bdf50-a414-4bf1-b21d-c94c5180ba24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:31:51.286593 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286425 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb6bdf50-a414-4bf1-b21d-c94c5180ba24" (UID: "bb6bdf50-a414-4bf1-b21d-c94c5180ba24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:31:51.286593 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286512 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-home" (OuterVolumeSpecName: "home") pod "bb6bdf50-a414-4bf1-b21d-c94c5180ba24" (UID: "bb6bdf50-a414-4bf1-b21d-c94c5180ba24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:31:51.286593 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286578 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-model-cache\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:31:51.286701 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.286601 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tmp-dir\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:31:51.288423 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.288390 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb6bdf50-a414-4bf1-b21d-c94c5180ba24" (UID: "bb6bdf50-a414-4bf1-b21d-c94c5180ba24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:31:51.288423 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.288404 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-kube-api-access-xpfpd" (OuterVolumeSpecName: "kube-api-access-xpfpd") pod "bb6bdf50-a414-4bf1-b21d-c94c5180ba24" (UID: "bb6bdf50-a414-4bf1-b21d-c94c5180ba24"). InnerVolumeSpecName "kube-api-access-xpfpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:31:51.288423 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.288393 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-dshm" (OuterVolumeSpecName: "dshm") pod "bb6bdf50-a414-4bf1-b21d-c94c5180ba24" (UID: "bb6bdf50-a414-4bf1-b21d-c94c5180ba24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:31:51.387968 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.387934 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpfpd\" (UniqueName: \"kubernetes.io/projected/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-kube-api-access-xpfpd\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:31:51.387968 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.387963 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-home\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:31:51.387968 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.387972 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-dshm\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:31:51.388193 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.387980 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb6bdf50-a414-4bf1-b21d-c94c5180ba24-tls-certs\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:31:51.463591 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.463551 2576 generic.go:356] "Generic (PLEG): container finished" podID="bb6bdf50-a414-4bf1-b21d-c94c5180ba24" containerID="722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185" exitCode=0 Jul 16 13:31:51.463591 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.463595 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" Jul 16 13:31:51.463792 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.463634 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" event={"ID":"bb6bdf50-a414-4bf1-b21d-c94c5180ba24","Type":"ContainerDied","Data":"722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185"} Jul 16 13:31:51.463792 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.463682 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99" event={"ID":"bb6bdf50-a414-4bf1-b21d-c94c5180ba24","Type":"ContainerDied","Data":"a63bc46da0500ed8801628af042a6e55b1c29f63f4fca7a3e4b0650ca79cca96"} Jul 16 13:31:51.463792 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.463699 2576 scope.go:117] "RemoveContainer" containerID="722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185" Jul 16 13:31:51.473245 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.473221 2576 scope.go:117] "RemoveContainer" containerID="722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185" Jul 16 13:31:51.473488 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:31:51.473462 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185\": container with ID starting with 722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185 not found: ID does not exist" containerID="722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185" Jul 16 13:31:51.473551 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.473488 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185"} err="failed to get container status \"722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185\": rpc error: code = NotFound desc = could not find container \"722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185\": container with ID starting with 722fd34df37bcb60d5f904852930ddfef47580c48979cec78e404993802b5185 not found: ID does not exist" Jul 16 13:31:51.500613 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.500590 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99"] Jul 16 13:31:51.505506 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:51.505486 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-c7dd05/sub-test-model-c7dd05-kserve-76fb6d6d65-m4n99"] Jul 16 13:31:52.986366 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:52.986332 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb6bdf50-a414-4bf1-b21d-c94c5180ba24" path="/var/lib/kubelet/pods/bb6bdf50-a414-4bf1-b21d-c94c5180ba24/volumes" Jul 16 13:31:54.711840 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:54.711787 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:54.712194 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:54.711850 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:54.724260 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:54.724238 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:31:55.492475 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:31:55.492448 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:32:23.694477 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:32:23.694436 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:32:51.999846 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:32:51.999801 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:33:06.103519 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:06.103445 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:33:13.304608 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:13.304573 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7"] Jul 16 13:33:15.257160 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.257125 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140680305363888"] Jul 16 13:33:15.257532 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.257451 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb6bdf50-a414-4bf1-b21d-c94c5180ba24" containerName="main" Jul 16 13:33:15.257532 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.257461 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb6bdf50-a414-4bf1-b21d-c94c5180ba24" containerName="main" Jul 16 13:33:15.257532 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.257527 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb6bdf50-a414-4bf1-b21d-c94c5180ba24" containerName="main" Jul 16 13:33:15.260351 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.260334 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305363888" Jul 16 13:33:15.264926 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.264903 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fd4h6\"" Jul 16 13:33:15.269318 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.269287 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305363888"] Jul 16 13:33:15.312557 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.312532 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwwbh\" (UniqueName: \"kubernetes.io/projected/cd38a30c-5087-47ac-aca1-246ffd040d1c-kube-api-access-jwwbh\") pod \"test-curl-7614-140680305363888\" (UID: \"cd38a30c-5087-47ac-aca1-246ffd040d1c\") " pod="opendatahub/test-curl-7614-140680305363888" Jul 16 13:33:15.413969 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.413940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwwbh\" (UniqueName: \"kubernetes.io/projected/cd38a30c-5087-47ac-aca1-246ffd040d1c-kube-api-access-jwwbh\") pod \"test-curl-7614-140680305363888\" (UID: \"cd38a30c-5087-47ac-aca1-246ffd040d1c\") " pod="opendatahub/test-curl-7614-140680305363888" Jul 16 13:33:15.425316 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.425291 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwwbh\" (UniqueName: \"kubernetes.io/projected/cd38a30c-5087-47ac-aca1-246ffd040d1c-kube-api-access-jwwbh\") pod \"test-curl-7614-140680305363888\" (UID: \"cd38a30c-5087-47ac-aca1-246ffd040d1c\") " pod="opendatahub/test-curl-7614-140680305363888" Jul 16 13:33:15.569966 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.569881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305363888" Jul 16 13:33:15.709317 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.709290 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305363888"] Jul 16 13:33:15.711389 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:33:15.711370 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd38a30c_5087_47ac_aca1_246ffd040d1c.slice/crio-240cb3cb87631eaed2362db365c6b9dcabaa1333aa90273eafa3d7da2e911d8a WatchSource:0}: Error finding container 240cb3cb87631eaed2362db365c6b9dcabaa1333aa90273eafa3d7da2e911d8a: Status 404 returned error can't find the container with id 240cb3cb87631eaed2362db365c6b9dcabaa1333aa90273eafa3d7da2e911d8a Jul 16 13:33:15.713088 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.713069 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 16 13:33:15.833601 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:15.833520 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305363888" event={"ID":"cd38a30c-5087-47ac-aca1-246ffd040d1c","Type":"ContainerStarted","Data":"240cb3cb87631eaed2362db365c6b9dcabaa1333aa90273eafa3d7da2e911d8a"} Jul 16 13:33:16.839646 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:16.839610 2576 generic.go:356] "Generic (PLEG): container finished" podID="cd38a30c-5087-47ac-aca1-246ffd040d1c" containerID="c9dff7409f7fcb4a657e53fb03f1dd9a094d1d5c7830c552cc2b929ea41240fa" exitCode=0 Jul 16 13:33:16.840047 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:16.839657 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305363888" event={"ID":"cd38a30c-5087-47ac-aca1-246ffd040d1c","Type":"ContainerDied","Data":"c9dff7409f7fcb4a657e53fb03f1dd9a094d1d5c7830c552cc2b929ea41240fa"} Jul 16 13:33:17.973621 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:17.973600 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305363888" Jul 16 13:33:18.025579 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.025554 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140680305363888_cd38a30c-5087-47ac-aca1-246ffd040d1c/test-curl-7614-140680305363888/0.log" Jul 16 13:33:18.036681 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.036658 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwwbh\" (UniqueName: \"kubernetes.io/projected/cd38a30c-5087-47ac-aca1-246ffd040d1c-kube-api-access-jwwbh\") pod \"cd38a30c-5087-47ac-aca1-246ffd040d1c\" (UID: \"cd38a30c-5087-47ac-aca1-246ffd040d1c\") " Jul 16 13:33:18.038797 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.038773 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd38a30c-5087-47ac-aca1-246ffd040d1c-kube-api-access-jwwbh" (OuterVolumeSpecName: "kube-api-access-jwwbh") pod "cd38a30c-5087-47ac-aca1-246ffd040d1c" (UID: "cd38a30c-5087-47ac-aca1-246ffd040d1c"). InnerVolumeSpecName "kube-api-access-jwwbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:33:18.057097 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.057074 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140680305363888"] Jul 16 13:33:18.062347 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.062328 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140680305363888"] Jul 16 13:33:18.137442 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.137378 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwwbh\" (UniqueName: \"kubernetes.io/projected/cd38a30c-5087-47ac-aca1-246ffd040d1c-kube-api-access-jwwbh\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:18.194178 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.194139 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140680305361712"] Jul 16 13:33:18.194524 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.194507 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd38a30c-5087-47ac-aca1-246ffd040d1c" containerName="test-curl-7614-140680305363888" Jul 16 13:33:18.194599 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.194528 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd38a30c-5087-47ac-aca1-246ffd040d1c" containerName="test-curl-7614-140680305363888" Jul 16 13:33:18.194660 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.194630 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd38a30c-5087-47ac-aca1-246ffd040d1c" containerName="test-curl-7614-140680305363888" Jul 16 13:33:18.197508 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.197490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:18.204766 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.204746 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305361712"] Jul 16 13:33:18.339073 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.339040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/16fa37f5-e6a8-4572-80a9-aa174291badd-kube-api-access-rnscj\") pod \"test-curl-7614-140680305361712\" (UID: \"16fa37f5-e6a8-4572-80a9-aa174291badd\") " pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:18.439684 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.439657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/16fa37f5-e6a8-4572-80a9-aa174291badd-kube-api-access-rnscj\") pod \"test-curl-7614-140680305361712\" (UID: \"16fa37f5-e6a8-4572-80a9-aa174291badd\") " pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:18.451724 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.451694 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/16fa37f5-e6a8-4572-80a9-aa174291badd-kube-api-access-rnscj\") pod \"test-curl-7614-140680305361712\" (UID: \"16fa37f5-e6a8-4572-80a9-aa174291badd\") " pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:18.507117 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.507090 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:18.636759 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.636731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305361712"] Jul 16 13:33:18.638452 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:33:18.638424 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16fa37f5_e6a8_4572_80a9_aa174291badd.slice/crio-aefc6ce40a97da7931376ab306184d0f1bc4a478a1c156eee4387438440e32d2 WatchSource:0}: Error finding container aefc6ce40a97da7931376ab306184d0f1bc4a478a1c156eee4387438440e32d2: Status 404 returned error can't find the container with id aefc6ce40a97da7931376ab306184d0f1bc4a478a1c156eee4387438440e32d2 Jul 16 13:33:18.849804 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.849718 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305363888" Jul 16 13:33:18.849804 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.849738 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="240cb3cb87631eaed2362db365c6b9dcabaa1333aa90273eafa3d7da2e911d8a" Jul 16 13:33:18.850936 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.850910 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305361712" event={"ID":"16fa37f5-e6a8-4572-80a9-aa174291badd","Type":"ContainerStarted","Data":"aefc6ce40a97da7931376ab306184d0f1bc4a478a1c156eee4387438440e32d2"} Jul 16 13:33:18.860652 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:18.860623 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305363888\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="cd38a30c-5087-47ac-aca1-246ffd040d1c" pod="opendatahub/test-curl-7614-140680305363888" Jul 16 13:33:18.986296 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:18.986272 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd38a30c-5087-47ac-aca1-246ffd040d1c" path="/var/lib/kubelet/pods/cd38a30c-5087-47ac-aca1-246ffd040d1c/volumes" Jul 16 13:33:19.856596 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:19.856559 2576 generic.go:356] "Generic (PLEG): container finished" podID="16fa37f5-e6a8-4572-80a9-aa174291badd" containerID="446edd4d7ebe5894062df14f9b9019fbb3777cab113999aba5c7e21e7eb2af7c" exitCode=0 Jul 16 13:33:19.856766 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:19.856644 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305361712" event={"ID":"16fa37f5-e6a8-4572-80a9-aa174291badd","Type":"ContainerDied","Data":"446edd4d7ebe5894062df14f9b9019fbb3777cab113999aba5c7e21e7eb2af7c"} Jul 16 13:33:20.205218 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:20.205184 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:33:20.993686 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:20.993665 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:21.020585 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.020556 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140680305361712_16fa37f5-e6a8-4572-80a9-aa174291badd/test-curl-7614-140680305361712/0.log" Jul 16 13:33:21.048466 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.048432 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140680305361712"] Jul 16 13:33:21.060875 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.060852 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/16fa37f5-e6a8-4572-80a9-aa174291badd-kube-api-access-rnscj\") pod \"16fa37f5-e6a8-4572-80a9-aa174291badd\" (UID: \"16fa37f5-e6a8-4572-80a9-aa174291badd\") " Jul 16 13:33:21.062926 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.062900 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fa37f5-e6a8-4572-80a9-aa174291badd-kube-api-access-rnscj" (OuterVolumeSpecName: "kube-api-access-rnscj") pod "16fa37f5-e6a8-4572-80a9-aa174291badd" (UID: "16fa37f5-e6a8-4572-80a9-aa174291badd"). InnerVolumeSpecName "kube-api-access-rnscj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:33:21.066469 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.066440 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140680305361712"] Jul 16 13:33:21.162182 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.162097 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnscj\" (UniqueName: \"kubernetes.io/projected/16fa37f5-e6a8-4572-80a9-aa174291badd-kube-api-access-rnscj\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:21.228467 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.228439 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:21.228807 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.228771 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16fa37f5-e6a8-4572-80a9-aa174291badd" containerName="test-curl-7614-140680305361712" Jul 16 13:33:21.228807 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.228782 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="16fa37f5-e6a8-4572-80a9-aa174291badd" containerName="test-curl-7614-140680305361712" Jul 16 13:33:21.228919 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.228855 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="16fa37f5-e6a8-4572-80a9-aa174291badd" containerName="test-curl-7614-140680305361712" Jul 16 13:33:21.231718 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.231704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:21.241981 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.241958 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:21.363334 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.363304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bbfd\" (UniqueName: \"kubernetes.io/projected/94ec031e-6da6-4bee-84eb-b57fc3d9beb1-kube-api-access-5bbfd\") pod \"test-curl-7614-140680305586512\" (UID: \"94ec031e-6da6-4bee-84eb-b57fc3d9beb1\") " pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:21.464573 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.464540 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bbfd\" (UniqueName: \"kubernetes.io/projected/94ec031e-6da6-4bee-84eb-b57fc3d9beb1-kube-api-access-5bbfd\") pod \"test-curl-7614-140680305586512\" (UID: \"94ec031e-6da6-4bee-84eb-b57fc3d9beb1\") " pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:21.474986 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.474959 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bbfd\" (UniqueName: \"kubernetes.io/projected/94ec031e-6da6-4bee-84eb-b57fc3d9beb1-kube-api-access-5bbfd\") pod \"test-curl-7614-140680305586512\" (UID: \"94ec031e-6da6-4bee-84eb-b57fc3d9beb1\") " pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:21.540156 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.540128 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:21.668639 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.668614 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:21.670631 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:33:21.670603 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94ec031e_6da6_4bee_84eb_b57fc3d9beb1.slice/crio-c1cfbd4782669e0da251227c2bf95d5d3dfefb7bcf0a627008c3a1791e41e342 WatchSource:0}: Error finding container c1cfbd4782669e0da251227c2bf95d5d3dfefb7bcf0a627008c3a1791e41e342: Status 404 returned error can't find the container with id c1cfbd4782669e0da251227c2bf95d5d3dfefb7bcf0a627008c3a1791e41e342 Jul 16 13:33:21.867777 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.867694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305586512" event={"ID":"94ec031e-6da6-4bee-84eb-b57fc3d9beb1","Type":"ContainerStarted","Data":"c1cfbd4782669e0da251227c2bf95d5d3dfefb7bcf0a627008c3a1791e41e342"} Jul 16 13:33:21.868885 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.868868 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aefc6ce40a97da7931376ab306184d0f1bc4a478a1c156eee4387438440e32d2" Jul 16 13:33:21.868987 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:21.868887 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:21.879663 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:21.879635 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305361712\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="16fa37f5-e6a8-4572-80a9-aa174291badd" pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:22.874392 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:22.874360 2576 generic.go:356] "Generic (PLEG): container finished" podID="94ec031e-6da6-4bee-84eb-b57fc3d9beb1" containerID="b8403db61e100740afbdd287f291fe2716ec26e7fd01f9f8911ba7766aec19da" exitCode=0 Jul 16 13:33:22.874806 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:22.874414 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305586512" event={"ID":"94ec031e-6da6-4bee-84eb-b57fc3d9beb1","Type":"ContainerDied","Data":"b8403db61e100740afbdd287f291fe2716ec26e7fd01f9f8911ba7766aec19da"} Jul 16 13:33:22.877428 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:22.877400 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305361712\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="16fa37f5-e6a8-4572-80a9-aa174291badd" pod="opendatahub/test-curl-7614-140680305361712" Jul 16 13:33:22.986218 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:22.986180 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16fa37f5-e6a8-4572-80a9-aa174291badd" path="/var/lib/kubelet/pods/16fa37f5-e6a8-4572-80a9-aa174291badd/volumes" Jul 16 13:33:24.006598 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.006574 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:24.030321 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.030297 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140680305586512_94ec031e-6da6-4bee-84eb-b57fc3d9beb1/test-curl-7614-140680305586512/0.log" Jul 16 13:33:24.055075 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.055050 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:24.059554 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.059533 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:24.085619 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.085596 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bbfd\" (UniqueName: \"kubernetes.io/projected/94ec031e-6da6-4bee-84eb-b57fc3d9beb1-kube-api-access-5bbfd\") pod \"94ec031e-6da6-4bee-84eb-b57fc3d9beb1\" (UID: \"94ec031e-6da6-4bee-84eb-b57fc3d9beb1\") " Jul 16 13:33:24.087661 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.087634 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ec031e-6da6-4bee-84eb-b57fc3d9beb1-kube-api-access-5bbfd" (OuterVolumeSpecName: "kube-api-access-5bbfd") pod "94ec031e-6da6-4bee-84eb-b57fc3d9beb1" (UID: "94ec031e-6da6-4bee-84eb-b57fc3d9beb1"). InnerVolumeSpecName "kube-api-access-5bbfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:33:24.187248 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.187212 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bbfd\" (UniqueName: \"kubernetes.io/projected/94ec031e-6da6-4bee-84eb-b57fc3d9beb1-kube-api-access-5bbfd\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:24.193361 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.193334 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:24.193705 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.193692 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ec031e-6da6-4bee-84eb-b57fc3d9beb1" containerName="test-curl-7614-140680305586512" Jul 16 13:33:24.193752 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.193707 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ec031e-6da6-4bee-84eb-b57fc3d9beb1" containerName="test-curl-7614-140680305586512" Jul 16 13:33:24.193798 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.193776 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ec031e-6da6-4bee-84eb-b57fc3d9beb1" containerName="test-curl-7614-140680305586512" Jul 16 13:33:24.196587 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.196571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:24.202720 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.202698 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:24.288478 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.288444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rbfx\" (UniqueName: \"kubernetes.io/projected/fd5e4442-01c1-4505-a178-48ee8b54b85e-kube-api-access-9rbfx\") pod \"test-curl-7614-140680305586512\" (UID: \"fd5e4442-01c1-4505-a178-48ee8b54b85e\") " pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:24.388910 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.388869 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rbfx\" (UniqueName: \"kubernetes.io/projected/fd5e4442-01c1-4505-a178-48ee8b54b85e-kube-api-access-9rbfx\") pod \"test-curl-7614-140680305586512\" (UID: \"fd5e4442-01c1-4505-a178-48ee8b54b85e\") " pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:24.399090 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.399060 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rbfx\" (UniqueName: \"kubernetes.io/projected/fd5e4442-01c1-4505-a178-48ee8b54b85e-kube-api-access-9rbfx\") pod \"test-curl-7614-140680305586512\" (UID: \"fd5e4442-01c1-4505-a178-48ee8b54b85e\") " pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:24.506276 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.506197 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:24.646883 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.646842 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:24.649217 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:33:24.649180 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd5e4442_01c1_4505_a178_48ee8b54b85e.slice/crio-1b99760b98b670f75de988e3908eac67fe214549fb7fd9c41b7c9db41df6a01a WatchSource:0}: Error finding container 1b99760b98b670f75de988e3908eac67fe214549fb7fd9c41b7c9db41df6a01a: Status 404 returned error can't find the container with id 1b99760b98b670f75de988e3908eac67fe214549fb7fd9c41b7c9db41df6a01a Jul 16 13:33:24.883625 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.883537 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1cfbd4782669e0da251227c2bf95d5d3dfefb7bcf0a627008c3a1791e41e342" Jul 16 13:33:24.883625 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.883538 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:24.884670 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.884647 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305586512" event={"ID":"fd5e4442-01c1-4505-a178-48ee8b54b85e","Type":"ContainerStarted","Data":"1b99760b98b670f75de988e3908eac67fe214549fb7fd9c41b7c9db41df6a01a"} Jul 16 13:33:24.895713 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.895685 2576 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7614-140680305586512" oldPodUID="94ec031e-6da6-4bee-84eb-b57fc3d9beb1" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" Jul 16 13:33:24.986689 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:24.986662 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ec031e-6da6-4bee-84eb-b57fc3d9beb1" path="/var/lib/kubelet/pods/94ec031e-6da6-4bee-84eb-b57fc3d9beb1/volumes" Jul 16 13:33:25.890490 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:25.890456 2576 generic.go:356] "Generic (PLEG): container finished" podID="fd5e4442-01c1-4505-a178-48ee8b54b85e" containerID="5e6c024b49eb898833790e9f02fe2a2f45291efd4ce9a5244774336b688d84cd" exitCode=0 Jul 16 13:33:25.890920 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:25.890546 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305586512" event={"ID":"fd5e4442-01c1-4505-a178-48ee8b54b85e","Type":"ContainerDied","Data":"5e6c024b49eb898833790e9f02fe2a2f45291efd4ce9a5244774336b688d84cd"} Jul 16 13:33:27.028755 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.028726 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:27.057076 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.057042 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140680305586512_fd5e4442-01c1-4505-a178-48ee8b54b85e/test-curl-7614-140680305586512/0.log" Jul 16 13:33:27.082274 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.082245 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:27.089361 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.089339 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140680305586512"] Jul 16 13:33:27.115395 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.115373 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rbfx\" (UniqueName: \"kubernetes.io/projected/fd5e4442-01c1-4505-a178-48ee8b54b85e-kube-api-access-9rbfx\") pod \"fd5e4442-01c1-4505-a178-48ee8b54b85e\" (UID: \"fd5e4442-01c1-4505-a178-48ee8b54b85e\") " Jul 16 13:33:27.117406 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.117384 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5e4442-01c1-4505-a178-48ee8b54b85e-kube-api-access-9rbfx" (OuterVolumeSpecName: "kube-api-access-9rbfx") pod "fd5e4442-01c1-4505-a178-48ee8b54b85e" (UID: "fd5e4442-01c1-4505-a178-48ee8b54b85e"). InnerVolumeSpecName "kube-api-access-9rbfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:33:27.216444 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.216409 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rbfx\" (UniqueName: \"kubernetes.io/projected/fd5e4442-01c1-4505-a178-48ee8b54b85e-kube-api-access-9rbfx\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:27.900516 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.900491 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b99760b98b670f75de988e3908eac67fe214549fb7fd9c41b7c9db41df6a01a" Jul 16 13:33:27.900516 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:27.900513 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:27.910501 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:27.910477 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305586512\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:28.323762 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.323726 2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" podUID="2898e66d-598b-4289-814c-4b54f109ea68" containerName="main" containerID="cri-o://a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc" gracePeriod=45 Jul 16 13:33:28.571670 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.571648 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:33:28.598253 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:28.597918 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305586512\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:28.730398 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730364 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-dshm\") pod \"2898e66d-598b-4289-814c-4b54f109ea68\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " Jul 16 13:33:28.730553 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730427 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-model-cache\") pod \"2898e66d-598b-4289-814c-4b54f109ea68\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " Jul 16 13:33:28.730553 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730460 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-home\") pod \"2898e66d-598b-4289-814c-4b54f109ea68\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " Jul 16 13:33:28.730553 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730474 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-tmp-dir\") pod \"2898e66d-598b-4289-814c-4b54f109ea68\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " Jul 16 13:33:28.730553 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730514 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6clk\" (UniqueName: \"kubernetes.io/projected/2898e66d-598b-4289-814c-4b54f109ea68-kube-api-access-g6clk\") pod \"2898e66d-598b-4289-814c-4b54f109ea68\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " Jul 16 13:33:28.730553 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730532 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2898e66d-598b-4289-814c-4b54f109ea68-tls-certs\") pod \"2898e66d-598b-4289-814c-4b54f109ea68\" (UID: \"2898e66d-598b-4289-814c-4b54f109ea68\") " Jul 16 13:33:28.730859 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730761 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2898e66d-598b-4289-814c-4b54f109ea68" (UID: "2898e66d-598b-4289-814c-4b54f109ea68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:33:28.730859 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730774 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-model-cache" (OuterVolumeSpecName: "model-cache") pod "2898e66d-598b-4289-814c-4b54f109ea68" (UID: "2898e66d-598b-4289-814c-4b54f109ea68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:33:28.730859 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.730793 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-home" (OuterVolumeSpecName: "home") pod "2898e66d-598b-4289-814c-4b54f109ea68" (UID: "2898e66d-598b-4289-814c-4b54f109ea68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:33:28.732554 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.732525 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-dshm" (OuterVolumeSpecName: "dshm") pod "2898e66d-598b-4289-814c-4b54f109ea68" (UID: "2898e66d-598b-4289-814c-4b54f109ea68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 16 13:33:28.732649 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.732581 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2898e66d-598b-4289-814c-4b54f109ea68-kube-api-access-g6clk" (OuterVolumeSpecName: "kube-api-access-g6clk") pod "2898e66d-598b-4289-814c-4b54f109ea68" (UID: "2898e66d-598b-4289-814c-4b54f109ea68"). InnerVolumeSpecName "kube-api-access-g6clk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:33:28.732692 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.732651 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2898e66d-598b-4289-814c-4b54f109ea68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2898e66d-598b-4289-814c-4b54f109ea68" (UID: "2898e66d-598b-4289-814c-4b54f109ea68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 16 13:33:28.832135 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.832095 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6clk\" (UniqueName: \"kubernetes.io/projected/2898e66d-598b-4289-814c-4b54f109ea68-kube-api-access-g6clk\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:28.832135 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.832130 2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2898e66d-598b-4289-814c-4b54f109ea68-tls-certs\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:28.832135 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.832141 2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-dshm\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:28.832390 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.832150 2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-model-cache\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:28.832390 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.832158 2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-home\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:28.832390 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.832186 2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2898e66d-598b-4289-814c-4b54f109ea68-tmp-dir\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:28.909986 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.909900 2576 generic.go:356] "Generic (PLEG): container finished" podID="2898e66d-598b-4289-814c-4b54f109ea68" containerID="a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc" exitCode=0 Jul 16 13:33:28.909986 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.909960 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" Jul 16 13:33:28.910175 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.909983 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" event={"ID":"2898e66d-598b-4289-814c-4b54f109ea68","Type":"ContainerDied","Data":"a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc"} Jul 16 13:33:28.910175 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.910026 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7" event={"ID":"2898e66d-598b-4289-814c-4b54f109ea68","Type":"ContainerDied","Data":"44a5067b69729d192dd75688147eb2bc5a508960a5012027800b790ce54f59ae"} Jul 16 13:33:28.910175 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.910044 2576 scope.go:117] "RemoveContainer" containerID="a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc" Jul 16 13:33:28.912910 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:28.912882 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305586512\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:28.918768 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.918747 2576 scope.go:117] "RemoveContainer" containerID="a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc" Jul 16 13:33:28.919061 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:28.919045 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc\": container with ID starting with a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc not found: ID does not exist" containerID="a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc" Jul 16 13:33:28.919119 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.919068 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc"} err="failed to get container status \"a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc\": rpc error: code = NotFound desc = could not find container \"a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc\": container with ID starting with a1c4cb728eadeedc6aa373ad0df07d47d6bc84459ab30607451f15ca8b9ea1fc not found: ID does not exist" Jul 16 13:33:28.920528 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:28.920506 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305586512\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" pod="opendatahub/test-curl-7614-140680305586512" Jul 16 13:33:28.936500 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.936478 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7"] Jul 16 13:33:28.944709 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.944688 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-978d92/rate-test-model-978d92-kserve-856f88c967-276g7"] Jul 16 13:33:28.986275 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.986244 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2898e66d-598b-4289-814c-4b54f109ea68" path="/var/lib/kubelet/pods/2898e66d-598b-4289-814c-4b54f109ea68/volumes" Jul 16 13:33:28.986589 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:28.986577 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" path="/var/lib/kubelet/pods/fd5e4442-01c1-4505-a178-48ee8b54b85e/volumes" Jul 16 13:33:30.775506 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.775471 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:30.775898 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.775800 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2898e66d-598b-4289-814c-4b54f109ea68" containerName="main" Jul 16 13:33:30.775898 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.775811 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2898e66d-598b-4289-814c-4b54f109ea68" containerName="main" Jul 16 13:33:30.775898 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.775835 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" containerName="test-curl-7614-140680305586512" Jul 16 13:33:30.775898 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.775841 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" containerName="test-curl-7614-140680305586512" Jul 16 13:33:30.775898 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.775898 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2898e66d-598b-4289-814c-4b54f109ea68" containerName="main" Jul 16 13:33:30.776063 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.775906 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd5e4442-01c1-4505-a178-48ee8b54b85e" containerName="test-curl-7614-140680305586512" Jul 16 13:33:30.778908 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.778891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:30.784328 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.784307 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fd4h6\"" Jul 16 13:33:30.790088 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.790068 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:30.848837 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.848784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfglz\" (UniqueName: \"kubernetes.io/projected/006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09-kube-api-access-qfglz\") pod \"test-curl-7614-140680305587520\" (UID: \"006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09\") " pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:30.950355 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.950313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfglz\" (UniqueName: \"kubernetes.io/projected/006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09-kube-api-access-qfglz\") pod \"test-curl-7614-140680305587520\" (UID: \"006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09\") " pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:30.960553 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:30.960523 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfglz\" (UniqueName: \"kubernetes.io/projected/006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09-kube-api-access-qfglz\") pod \"test-curl-7614-140680305587520\" (UID: \"006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09\") " pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:31.088085 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:31.088009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:31.220953 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:31.220904 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:31.223589 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:33:31.223563 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod006f75c8_bc4f_4bd9_b7eb_699bf0b3bd09.slice/crio-a3a17923eca552bcbcc1d3b5da4fe9acbaaf51b550e3512e0d84d3592b907070 WatchSource:0}: Error finding container a3a17923eca552bcbcc1d3b5da4fe9acbaaf51b550e3512e0d84d3592b907070: Status 404 returned error can't find the container with id a3a17923eca552bcbcc1d3b5da4fe9acbaaf51b550e3512e0d84d3592b907070 Jul 16 13:33:31.928454 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:31.928420 2576 generic.go:356] "Generic (PLEG): container finished" podID="006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09" containerID="9a876f64d1569cfb987335690addc4f25dbd94f10f03b8450307f6af90630e92" exitCode=0 Jul 16 13:33:31.928843 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:31.928496 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305587520" event={"ID":"006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09","Type":"ContainerDied","Data":"9a876f64d1569cfb987335690addc4f25dbd94f10f03b8450307f6af90630e92"} Jul 16 13:33:31.928843 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:31.928523 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7614-140680305587520" event={"ID":"006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09","Type":"ContainerStarted","Data":"a3a17923eca552bcbcc1d3b5da4fe9acbaaf51b550e3512e0d84d3592b907070"} Jul 16 13:33:33.066108 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.066086 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:33.096088 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.096066 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7614-140680305587520_006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09/test-curl-7614-140680305587520/0.log" Jul 16 13:33:33.121910 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.121884 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:33.124533 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.124509 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7614-140680305587520"] Jul 16 13:33:33.169863 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.169841 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfglz\" (UniqueName: \"kubernetes.io/projected/006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09-kube-api-access-qfglz\") pod \"006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09\" (UID: \"006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09\") " Jul 16 13:33:33.171877 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.171846 2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09-kube-api-access-qfglz" (OuterVolumeSpecName: "kube-api-access-qfglz") pod "006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09" (UID: "006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09"). InnerVolumeSpecName "kube-api-access-qfglz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 16 13:33:33.271447 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.271370 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfglz\" (UniqueName: \"kubernetes.io/projected/006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09-kube-api-access-qfglz\") on node \"ip-10-0-139-160.ec2.internal\" DevicePath \"\"" Jul 16 13:33:33.937625 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.937592 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:33.937772 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:33.937601 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3a17923eca552bcbcc1d3b5da4fe9acbaaf51b550e3512e0d84d3592b907070" Jul 16 13:33:33.948580 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:33.948549 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305587520\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:34.987554 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:34.987523 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09" path="/var/lib/kubelet/pods/006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09/volumes" Jul 16 13:33:34.989717 ip-10-0-139-160 kubenswrapper[2576]: E0716 13:33:34.989693 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7614-140680305587520\" is forbidden: User \"system:node:ip-10-0-139-160.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-160.ec2.internal' and this object" podUID="006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09" pod="opendatahub/test-curl-7614-140680305587520" Jul 16 13:33:47.596643 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:33:47.596601 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:34:13.908116 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:13.908080 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xsdkx"] Jul 16 13:34:26.138960 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:26.138910 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c8f755c7-wsxk6_f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a/authorino/0.log" Jul 16 13:34:32.440692 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:32.440662 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9d668944c-dckbb_ce83b313-fb7f-43fa-94be-9bb05e047ef6/copy-manifests/0.log" Jul 16 13:34:32.460789 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:32.460755 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9d668944c-dckbb_ce83b313-fb7f-43fa-94be-9bb05e047ef6/manager/0.log" Jul 16 13:34:36.420968 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:36.420864 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c8f755c7-wsxk6_f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a/authorino/0.log" Jul 16 13:34:36.812638 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:36.812559 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-w8g8l_b633b4c9-328f-4b8c-8be6-f7d79c481c66/kuadrant-console-plugin/0.log" Jul 16 13:34:37.215107 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:37.215079 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xsdkx_b36329e8-ad97-46cd-978e-4dd3f86de5a6/limitador/0.log" Jul 16 13:34:37.992170 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:37.992142 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d67ccd5b7-5xxlg_ab3f9b62-2def-4699-bd87-c3bb09c93010/kube-auth-proxy/0.log" Jul 16 13:34:39.515085 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:39.515042 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-2xct4_a53b1eb9-e4ab-448d-b41e-2e2572e25aae/main/0.log" Jul 16 13:34:39.642166 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:39.642128 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-dwzfj_bc4b2901-1652-4716-a40e-30054c7fcf92/main/0.log" Jul 16 13:34:49.133466 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:49.133434 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-pvcrz_0b6659fe-fabb-4a30-aaba-3615b0e73f11/global-pull-secret-syncer/0.log" Jul 16 13:34:49.422987 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:49.422954 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gzltq_fc0dd1b8-c7e2-44a2-b653-135b66d1654b/konnectivity-agent/0.log" Jul 16 13:34:49.584837 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:49.584787 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-160.ec2.internal_0bbc06f37afebb87bdba8d241d51d6dd/haproxy/0.log" Jul 16 13:34:54.900369 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:54.900333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-56c8f755c7-wsxk6_f9d1b59c-a40d-4bc0-a686-e8eeb8f4ec4a/authorino/0.log" Jul 16 13:34:55.021898 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:55.021802 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-w8g8l_b633b4c9-328f-4b8c-8be6-f7d79c481c66/kuadrant-console-plugin/0.log" Jul 16 13:34:55.266519 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:55.266492 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xsdkx_b36329e8-ad97-46cd-978e-4dd3f86de5a6/limitador/0.log" Jul 16 13:34:57.386577 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:57.386552 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-2dtkn_7325a195-047d-44d5-86e0-31c918f6a9ea/kube-state-metrics/0.log" Jul 16 13:34:57.437862 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:57.437839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-2dtkn_7325a195-047d-44d5-86e0-31c918f6a9ea/kube-rbac-proxy-main/0.log" Jul 16 13:34:57.486170 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:57.486134 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-2dtkn_7325a195-047d-44d5-86e0-31c918f6a9ea/kube-rbac-proxy-self/0.log" Jul 16 13:34:58.088976 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.088947 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnb6q_f5b36e44-0e72-4ec2-b640-ab003f3dfa6a/node-exporter/0.log" Jul 16 13:34:58.183785 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.183757 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnb6q_f5b36e44-0e72-4ec2-b640-ab003f3dfa6a/kube-rbac-proxy/0.log" Jul 16 13:34:58.241856 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.241832 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hnb6q_f5b36e44-0e72-4ec2-b640-ab003f3dfa6a/init-textfile/0.log" Jul 16 13:34:58.539439 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.539411 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b12af3b5-3373-447b-9103-15a422c067e9/prometheus/0.log" Jul 16 13:34:58.602654 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.602629 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b12af3b5-3373-447b-9103-15a422c067e9/config-reloader/0.log" Jul 16 13:34:58.672523 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.672496 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b12af3b5-3373-447b-9103-15a422c067e9/thanos-sidecar/0.log" Jul 16 13:34:58.735524 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.735497 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b12af3b5-3373-447b-9103-15a422c067e9/kube-rbac-proxy-web/0.log" Jul 16 13:34:58.828418 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.828336 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b12af3b5-3373-447b-9103-15a422c067e9/kube-rbac-proxy/0.log" Jul 16 13:34:58.892571 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.892515 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b12af3b5-3373-447b-9103-15a422c067e9/kube-rbac-proxy-thanos/0.log" Jul 16 13:34:58.944221 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:58.944179 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b12af3b5-3373-447b-9103-15a422c067e9/init-config-reloader/0.log" Jul 16 13:34:59.222218 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.222190 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-557ff67dcd-znxkl_e1b7bfa2-5769-486b-af68-4bb2868f7264/telemeter-client/0.log" Jul 16 13:34:59.273385 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.273358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-557ff67dcd-znxkl_e1b7bfa2-5769-486b-af68-4bb2868f7264/reload/0.log" Jul 16 13:34:59.334651 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.334628 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-557ff67dcd-znxkl_e1b7bfa2-5769-486b-af68-4bb2868f7264/kube-rbac-proxy/0.log" Jul 16 13:34:59.666812 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.666739 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628"] Jul 16 13:34:59.667169 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.667090 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09" containerName="test-curl-7614-140680305587520" Jul 16 13:34:59.667169 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.667101 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09" containerName="test-curl-7614-140680305587520" Jul 16 13:34:59.667169 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.667150 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="006f75c8-bc4f-4bd9-b7eb-699bf0b3bd09" containerName="test-curl-7614-140680305587520" Jul 16 13:34:59.670128 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.670104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.686114 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.686092 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ljjhv\"/\"openshift-service-ca.crt\"" Jul 16 13:34:59.686222 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.686098 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ljjhv\"/\"kube-root-ca.crt\"" Jul 16 13:34:59.686222 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.686101 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ljjhv\"/\"default-dockercfg-t5vwb\"" Jul 16 13:34:59.706025 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.705995 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628"] Jul 16 13:34:59.714750 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.714722 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-lib-modules\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.714847 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.714784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-podres\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.714902 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.714854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqqm8\" (UniqueName: \"kubernetes.io/projected/d78e89a8-af12-41b3-8d55-e891282adade-kube-api-access-hqqm8\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.714945 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.714921 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-sys\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.714990 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.714954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-proc\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.815971 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.815935 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-podres\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.815971 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.815973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqqm8\" (UniqueName: \"kubernetes.io/projected/d78e89a8-af12-41b3-8d55-e891282adade-kube-api-access-hqqm8\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.816182 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.815998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-sys\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.816182 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.816019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-proc\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.816182 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.816071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-lib-modules\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.816182 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.816091 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-sys\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.816182 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.816105 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-podres\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.816341 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.816212 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-lib-modules\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.816341 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.816214 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d78e89a8-af12-41b3-8d55-e891282adade-proc\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.827781 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.827749 2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqqm8\" (UniqueName: \"kubernetes.io/projected/d78e89a8-af12-41b3-8d55-e891282adade-kube-api-access-hqqm8\") pod \"perf-node-gather-daemonset-p4628\" (UID: \"d78e89a8-af12-41b3-8d55-e891282adade\") " pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:34:59.979422 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:34:59.979381 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:35:00.123579 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:00.123545 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628"] Jul 16 13:35:00.128176 ip-10-0-139-160 kubenswrapper[2576]: W0716 13:35:00.128151 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd78e89a8_af12_41b3_8d55_e891282adade.slice/crio-e3340941b6414a94ce1fb6ebc1553f83bebad8900c5e702f6d2f456d2c1b4ba2 WatchSource:0}: Error finding container e3340941b6414a94ce1fb6ebc1553f83bebad8900c5e702f6d2f456d2c1b4ba2: Status 404 returned error can't find the container with id e3340941b6414a94ce1fb6ebc1553f83bebad8900c5e702f6d2f456d2c1b4ba2 Jul 16 13:35:00.334032 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:00.333998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" event={"ID":"d78e89a8-af12-41b3-8d55-e891282adade","Type":"ContainerStarted","Data":"5e34011cc43abf7e129889157211813136d7e8e8aa92f3fd04cfea6dce2323da"} Jul 16 13:35:00.334194 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:00.334050 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" event={"ID":"d78e89a8-af12-41b3-8d55-e891282adade","Type":"ContainerStarted","Data":"e3340941b6414a94ce1fb6ebc1553f83bebad8900c5e702f6d2f456d2c1b4ba2"} Jul 16 13:35:00.334194 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:00.334127 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:35:01.201796 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:01.201763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-gdrxd_5939f4ee-ca79-4129-9937-fd75490ec6f2/networking-console-plugin/0.log" Jul 16 13:35:01.916253 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:01.916228 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:35:01.921525 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:01.921505 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/2.log" Jul 16 13:35:03.439317 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:03.439289 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-gn492_db1a9477-cd53-41f9-b36c-b9831895cca0/volume-data-source-validator/0.log" Jul 16 13:35:05.074191 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:05.074161 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8l5pc_06467239-8bf8-4732-a273-e283e67af2dc/dns/0.log" Jul 16 13:35:05.159165 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:05.159136 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8l5pc_06467239-8bf8-4732-a273-e283e67af2dc/kube-rbac-proxy/0.log" Jul 16 13:35:05.390404 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:05.390329 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l764j_4642c698-57ae-49ad-a5f5-54c575f0fde9/dns-node-resolver/0.log" Jul 16 13:35:06.118337 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:06.118309 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-69d9478f48-wlfkb_210378ab-b60e-47ad-8751-32ad7feaff6f/registry/0.log" Jul 16 13:35:06.310840 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:06.310795 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztrvw_5664fccb-777b-4367-8239-458af04f6c29/node-ca/0.log" Jul 16 13:35:06.348979 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:06.348956 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" Jul 16 13:35:06.382167 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:06.382073 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ljjhv/perf-node-gather-daemonset-p4628" podStartSLOduration=7.382057823 podStartE2EDuration="7.382057823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 13:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 13:35:00.383026138 +0000 UTC m=+2985.925632884" watchObservedRunningTime="2026-07-16 13:35:06.382057823 +0000 UTC m=+2991.924664570" Jul 16 13:35:07.767557 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:07.767526 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d67ccd5b7-5xxlg_ab3f9b62-2def-4699-bd87-c3bb09c93010/kube-auth-proxy/0.log" Jul 16 13:35:09.085117 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:09.085088 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-f4c78_38cf452b-ac67-4fcb-bfa3-312e08ac1687/serve-healthcheck-canary/0.log" Jul 16 13:35:09.612226 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:09.612195 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d67ccd5b7-5xxlg_ab3f9b62-2def-4699-bd87-c3bb09c93010/kube-auth-proxy/0.log" Jul 16 13:35:10.133881 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:10.133848 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9s9qn_852279b4-b62e-4245-8859-e0baa4f102a0/kube-rbac-proxy/0.log" Jul 16 13:35:10.192374 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:10.192348 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9s9qn_852279b4-b62e-4245-8859-e0baa4f102a0/exporter/0.log" Jul 16 13:35:10.249909 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:10.249887 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9s9qn_852279b4-b62e-4245-8859-e0baa4f102a0/extractor/0.log" Jul 16 13:35:11.195130 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:11.195110 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d67ccd5b7-5xxlg_ab3f9b62-2def-4699-bd87-c3bb09c93010/kube-auth-proxy/0.log" Jul 16 13:35:13.810321 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:13.810292 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9d668944c-dckbb_ce83b313-fb7f-43fa-94be-9bb05e047ef6/manager/0.log" Jul 16 13:35:13.877874 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:13.877839 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9d668944c-dckbb_ce83b313-fb7f-43fa-94be-9bb05e047ef6/copy-manifests/0.log" Jul 16 13:35:15.139262 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:15.139225 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:35:15.146006 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:15.145984 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:35:15.148217 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:15.148194 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-dswpk_f1441a03-825b-4cb2-8d29-35bfd66d04f9/console-operator/1.log" Jul 16 13:35:15.155897 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:15.155876 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d62t6_caf2381b-6027-4b9d-bea9-f20120aec027/ovn-acl-logging/0.log" Jul 16 13:35:15.247309 ip-10-0-139-160 kubenswrapper[2576]: I0716 13:35:15.247280 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5864f4f95b-nfwxg_49771ff3-0caf-4106-aa7c-fd31f63bc483/manager/0.log"