Sep 16 18:45:12.332158 ip-10-0-142-175 systemd[1]: Starting Kubernetes Kubelet... Sep 16 18:45:12.766551 ip-10-0-142-175 kubenswrapper[2570]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:12.766551 ip-10-0-142-175 kubenswrapper[2570]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 18:45:12.766551 ip-10-0-142-175 kubenswrapper[2570]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:12.766551 ip-10-0-142-175 kubenswrapper[2570]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 18:45:12.766551 ip-10-0-142-175 kubenswrapper[2570]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:12.768067 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.767936 2570 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772066 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772085 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772089 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772092 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772095 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772099 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772102 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772105 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772108 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772111 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.772094 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772115 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772118 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772121 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772123 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772126 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772128 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772131 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772134 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772137 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772139 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772142 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772144 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772147 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772150 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772152 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772156 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772159 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772162 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772165 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772168 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.773818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772170 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772173 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772175 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772178 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772180 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772183 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772186 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772188 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772191 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772193 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772196 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772198 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772201 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772203 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772206 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772210 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772214 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772219 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772222 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.774352 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772225 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772228 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772230 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772233 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772235 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772238 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772241 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772243 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772245 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772248 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772251 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772254 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772256 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772259 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772261 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772264 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772266 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772269 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772309 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772312 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.774827 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772315 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772318 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772320 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772324 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772326 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772329 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772331 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772335 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772338 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772341 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772344 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772346 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772349 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772351 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772354 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772358 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772361 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772364 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772367 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.775322 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772370 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772373 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772375 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772378 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772380 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772384 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772388 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772390 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772393 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772397 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.775818 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.772401 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775927 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775936 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775940 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775944 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775947 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775950 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775953 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775955 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775958 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775961 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775963 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775966 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775968 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775971 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775973 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775976 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775978 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775981 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775983 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.776077 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775986 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775988 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775991 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775993 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775995 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.775998 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776000 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776003 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776006 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776009 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776012 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776014 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776017 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776019 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776022 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776025 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776027 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776029 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776032 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776034 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.776575 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776036 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776039 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776042 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776044 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776047 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776051 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776055 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776059 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776062 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776065 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776068 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776070 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776073 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776075 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776078 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776081 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776084 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776086 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776089 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776092 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.777115 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776094 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776097 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776100 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776102 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776105 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776108 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776112 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776116 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776118 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776121 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776123 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776126 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776128 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776131 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776133 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776135 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776138 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776140 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776143 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776145 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.777607 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776147 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776150 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776152 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776156 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776159 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776161 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776164 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776166 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776169 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776171 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776174 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776176 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776182 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776185 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776188 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776191 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776193 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776197 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776199 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.778110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776203 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776611 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776616 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776620 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776623 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776626 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776629 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776632 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776635 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776638 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776641 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776643 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776647 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776650 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776652 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776655 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776658 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776660 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776663 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776666 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.778664 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776669 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776671 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776673 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776676 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776678 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776682 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776686 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776689 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776693 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776696 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776700 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776703 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776706 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776708 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776711 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776714 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776717 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776719 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776724 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.779177 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776726 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776729 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776732 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776734 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776736 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776739 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776742 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776744 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776747 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776749 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776752 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776756 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776759 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776761 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776764 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776766 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776769 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776772 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776774 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.779645 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776776 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776779 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776782 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776784 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776787 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776790 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776792 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776796 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776798 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776801 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776803 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776806 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776808 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776810 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776813 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776815 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776818 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776820 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776822 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776825 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.780109 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776827 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776830 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776832 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776834 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776837 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776840 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776842 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776845 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776847 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776849 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776852 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776854 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776857 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776877 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776880 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776883 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776886 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776888 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776891 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776908 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.780588 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776912 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.776914 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.776986 2570 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777020 2570 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777026 2570 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777031 2570 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777036 2570 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777040 2570 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777043 2570 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777048 2570 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777051 2570 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777054 2570 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777058 2570 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777061 2570 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777065 2570 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777068 2570 flags.go:64] FLAG: --cgroup-root="" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777071 2570 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777074 2570 flags.go:64] FLAG: --client-ca-file="" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777077 2570 flags.go:64] FLAG: --cloud-provider="external" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777079 2570 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777091 2570 flags.go:64] FLAG: --cluster-domain="" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777094 2570 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777097 2570 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777100 2570 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 18:45:12.781095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777103 2570 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777107 2570 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777110 2570 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777113 2570 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777118 2570 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777122 2570 flags.go:64] FLAG: --contention-profiling="false" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777125 2570 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777128 2570 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777131 2570 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777134 2570 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777138 2570 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777141 2570 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777144 2570 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777147 2570 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777150 2570 flags.go:64] FLAG: --enable-server="true" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777153 2570 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777160 2570 flags.go:64] FLAG: --event-burst="100" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777163 2570 flags.go:64] FLAG: --event-qps="50" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777165 2570 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777169 2570 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777172 2570 flags.go:64] FLAG: --eviction-hard="" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777176 2570 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777179 2570 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777182 2570 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777185 2570 flags.go:64] FLAG: --eviction-soft="" Sep 16 18:45:12.781670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777190 2570 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777193 2570 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777196 2570 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777199 2570 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777202 2570 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777205 2570 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777207 2570 flags.go:64] FLAG: --feature-gates="" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777211 2570 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777214 2570 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777217 2570 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777220 2570 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777224 2570 flags.go:64] FLAG: --healthz-port="10248" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777228 2570 flags.go:64] FLAG: --help="false" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777231 2570 flags.go:64] FLAG: --hostname-override="ip-10-0-142-175.ec2.internal" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777234 2570 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777237 2570 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777240 2570 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777243 2570 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777246 2570 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777249 2570 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777252 2570 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777254 2570 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777257 2570 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 18:45:12.782317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777260 2570 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777263 2570 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777266 2570 flags.go:64] FLAG: --kube-reserved="" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777269 2570 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777272 2570 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777274 2570 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777277 2570 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777280 2570 flags.go:64] FLAG: --lock-file="" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777283 2570 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777287 2570 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777290 2570 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777295 2570 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777298 2570 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777301 2570 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777303 2570 flags.go:64] FLAG: --logging-format="text" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777306 2570 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777309 2570 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777312 2570 flags.go:64] FLAG: --manifest-url="" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777315 2570 flags.go:64] FLAG: --manifest-url-header="" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777319 2570 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777323 2570 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777328 2570 flags.go:64] FLAG: --max-pods="110" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777331 2570 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777334 2570 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777337 2570 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 18:45:12.782912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777340 2570 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777343 2570 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777346 2570 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777349 2570 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777356 2570 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777359 2570 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777362 2570 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777365 2570 flags.go:64] FLAG: --pod-cidr="" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777368 2570 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777373 2570 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777376 2570 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777379 2570 flags.go:64] FLAG: --pods-per-core="0" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777382 2570 flags.go:64] FLAG: --port="10250" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777385 2570 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777387 2570 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0bf12bb60be2ae0d7" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777390 2570 flags.go:64] FLAG: --qos-reserved="" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777395 2570 flags.go:64] FLAG: --read-only-port="10255" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777397 2570 flags.go:64] FLAG: --register-node="true" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777400 2570 flags.go:64] FLAG: --register-with-taints="" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777404 2570 flags.go:64] FLAG: --registry-burst="10" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777406 2570 flags.go:64] FLAG: --registry-qps="5" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777409 2570 flags.go:64] FLAG: --reserved-cpus="" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777412 2570 flags.go:64] FLAG: --reserved-memory="" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777416 2570 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777419 2570 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 18:45:12.783521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777421 2570 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777424 2570 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777427 2570 flags.go:64] FLAG: --runonce="false" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777431 2570 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777434 2570 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777437 2570 flags.go:64] FLAG: --seccomp-default="false" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777440 2570 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777443 2570 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777446 2570 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777449 2570 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777452 2570 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777454 2570 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777457 2570 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777460 2570 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777463 2570 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777466 2570 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777469 2570 flags.go:64] FLAG: --system-cgroups="" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777471 2570 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777477 2570 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777480 2570 flags.go:64] FLAG: --tls-cert-file="" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777483 2570 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777487 2570 flags.go:64] FLAG: --tls-min-version="" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777490 2570 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777494 2570 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777497 2570 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 18:45:12.784166 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777499 2570 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777502 2570 flags.go:64] FLAG: --v="2" Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777506 2570 flags.go:64] FLAG: --version="false" Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777510 2570 flags.go:64] FLAG: --vmodule="" Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777514 2570 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.777517 2570 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777641 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777644 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777647 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777650 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777654 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777657 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777660 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777663 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777666 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777668 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777671 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777673 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777676 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777679 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777681 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.784782 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777684 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777686 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777690 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777694 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777697 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777700 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777702 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777705 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777707 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777712 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777714 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777717 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777719 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777721 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777724 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777726 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777729 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777731 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777734 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.785315 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777736 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777739 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777742 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777745 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777747 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777750 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777753 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777756 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777758 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777761 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777764 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777766 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777769 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777771 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777774 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777777 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777779 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777782 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777784 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777787 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.785791 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777789 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777792 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777795 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777798 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777800 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777803 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777805 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777808 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777810 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777813 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777816 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777818 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777821 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777823 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777828 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777830 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777833 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777837 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777840 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.786300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777844 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777847 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777850 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777852 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777855 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777873 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777877 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777879 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777882 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777886 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777889 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777891 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777893 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777896 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777899 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777903 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777905 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777908 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777919 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777922 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.786766 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777924 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777927 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777931 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777934 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777937 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.777941 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.778830 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.786150 2570 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.786168 2570 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.786182 2570 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.786187 2570 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786244 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786250 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786253 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.787261 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786256 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786259 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786262 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786264 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786267 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786270 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786272 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786275 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786277 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786280 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786282 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786285 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786287 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786291 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786294 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786296 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786299 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786302 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786305 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786307 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.787639 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786310 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786313 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786316 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786318 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786321 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786323 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786326 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786329 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786332 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786335 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786337 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786340 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786343 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786345 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786348 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786350 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786353 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786356 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786358 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786360 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.788927 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786363 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786365 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786368 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786371 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786373 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786377 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786381 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786384 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786387 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786390 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786393 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786395 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786398 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786400 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786403 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786405 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786408 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786410 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786413 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786416 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.789516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786418 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786421 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786423 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786425 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786428 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786431 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786434 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786437 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786439 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786441 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786445 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786448 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786451 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786454 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786457 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786460 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786463 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786465 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786468 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.790083 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786470 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786472 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786475 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786478 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786480 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786483 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786485 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786488 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786491 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786494 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786498 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786501 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786504 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786506 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786509 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786512 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.790571 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786514 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.786519 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786638 2570 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786643 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786646 2570 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786649 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786652 2570 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786655 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786657 2570 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786660 2570 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786663 2570 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786666 2570 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786669 2570 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786671 2570 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.790981 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786674 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786677 2570 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786681 2570 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786684 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786686 2570 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786689 2570 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786691 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786694 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786696 2570 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786699 2570 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786701 2570 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786703 2570 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786706 2570 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786708 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786711 2570 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786714 2570 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786716 2570 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786718 2570 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786721 2570 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786723 2570 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.791319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786726 2570 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786728 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786731 2570 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786735 2570 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786738 2570 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786741 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786744 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786747 2570 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786749 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786753 2570 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786755 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786758 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786761 2570 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786763 2570 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786766 2570 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786768 2570 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786771 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786773 2570 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786776 2570 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.791847 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786779 2570 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786781 2570 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786784 2570 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786786 2570 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786789 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786791 2570 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786794 2570 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786797 2570 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786799 2570 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786802 2570 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786805 2570 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786807 2570 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786809 2570 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786812 2570 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786814 2570 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786817 2570 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786820 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786822 2570 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786824 2570 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786827 2570 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.792319 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786829 2570 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786832 2570 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786835 2570 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786837 2570 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786840 2570 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786842 2570 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786844 2570 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786847 2570 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786849 2570 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786852 2570 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786854 2570 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786857 2570 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786873 2570 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786876 2570 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786879 2570 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786881 2570 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786884 2570 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786887 2570 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786890 2570 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786892 2570 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.792794 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786895 2570 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786898 2570 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786901 2570 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786903 2570 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786906 2570 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786908 2570 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786911 2570 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:12.786914 2570 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.786920 2570 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.787038 2570 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.790244 2570 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.791255 2570 server.go:1019] "Starting client certificate rotation" Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.791352 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:45:12.793391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.791386 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:45:12.793960 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.793946 2570 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:45:12.797397 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.797376 2570 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:45:12.801830 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.801812 2570 log.go:25] "Validated CRI v1 runtime API" Sep 16 18:45:12.804725 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.804705 2570 log.go:25] "Validated CRI v1 image API" Sep 16 18:45:12.805676 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.805650 2570 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 18:45:12.809784 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.809765 2570 fs.go:135] Filesystem UUIDs: map[6b6f9d35-adab-479e-9840-edae64f6aa25:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9a99588d-9e15-447f-8c0d-be75590a7dd9:/dev/nvme0n1p3] Sep 16 18:45:12.809846 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.809784 2570 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 18:45:12.815216 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.815106 2570 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:12.813495262 +0000 UTC m=+0.356801012 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2499998 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22349f6089d4a1e5f09c56ecbd654f SystemUUID:ec22349f-6089-d4a1-e5f0-9c56ecbd654f BootID:bd3a4b01-fd37-484c-bc71-68fdfe55a0db Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:ff:12:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:ff:12:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:0a:03:0a:00:4b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 18:45:12.815216 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.815213 2570 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 18:45:12.815351 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.815328 2570 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 18:45:12.819094 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.819060 2570 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 18:45:12.819257 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.819097 2570 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-175.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 18:45:12.819947 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.819936 2570 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 18:45:12.819981 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.819949 2570 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 18:45:12.819981 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.819957 2570 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:45:12.820586 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.820576 2570 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:45:12.821474 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.821465 2570 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 18:45:12.825771 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.825758 2570 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:45:12.825911 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.825902 2570 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 18:45:12.825952 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.825936 2570 kubelet.go:486] "Attempting to sync node with API server" Sep 16 18:45:12.825952 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.825951 2570 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 18:45:12.826003 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.825963 2570 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 18:45:12.826003 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.825972 2570 kubelet.go:398] "Adding apiserver pod source" Sep 16 18:45:12.826003 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.825980 2570 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 18:45:12.826622 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.826606 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:45:12.826678 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.826634 2570 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:45:12.830838 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.830819 2570 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 18:45:12.832971 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.832956 2570 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 18:45:12.833031 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.832986 2570 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 18:45:12.834836 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.834821 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:45:12.835167 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835154 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 18:45:12.835199 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835177 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 18:45:12.835199 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835183 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 18:45:12.835199 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835189 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 18:45:12.835199 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835197 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 18:45:12.835311 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835203 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 18:45:12.835311 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835218 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 18:45:12.835311 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835223 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 18:45:12.835311 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835230 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 18:45:12.835311 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835235 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 18:45:12.835311 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835243 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 18:45:12.835311 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.835249 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 18:45:12.836885 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.836855 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 18:45:12.836885 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.836885 2570 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 18:45:12.839313 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.839287 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 18:45:12.839386 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.839332 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 18:45:12.840925 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.840912 2570 server.go:1268] "Started kubelet" Sep 16 18:45:12.841043 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.841018 2570 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 18:45:12.841100 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.841008 2570 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 18:45:12.841145 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.841102 2570 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 18:45:12.841739 ip-10-0-142-175 systemd[1]: Started Kubernetes Kubelet. Sep 16 18:45:12.843047 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.842925 2570 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 18:45:12.843239 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.843225 2570 server.go:310] "Adding debug handlers to kubelet server" Sep 16 18:45:12.845444 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.845424 2570 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 18:45:12.847589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.847575 2570 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 18:45:12.848184 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.848166 2570 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 18:45:12.848373 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.848321 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:12.848814 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.848794 2570 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 18:45:12.848814 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.848804 2570 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 18:45:12.848982 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.848820 2570 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 18:45:12.848982 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.848978 2570 reconstruct.go:97] "Volume reconstruction finished" Sep 16 18:45:12.849075 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.848986 2570 reconciler.go:29] "Reconciler: start to sync state" Sep 16 18:45:12.849143 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.849129 2570 factory.go:55] Registering systemd factory Sep 16 18:45:12.849193 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.849148 2570 factory.go:223] Registration of the systemd container factory successfully Sep 16 18:45:12.849530 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.849517 2570 factory.go:153] Registering CRI-O factory Sep 16 18:45:12.849568 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.849533 2570 factory.go:223] Registration of the crio container factory successfully Sep 16 18:45:12.849601 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.849570 2570 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 18:45:12.849601 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.849584 2570 factory.go:103] Registering Raw factory Sep 16 18:45:12.849601 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.849594 2570 manager.go:1196] Started watching for new ooms in manager Sep 16 18:45:12.850071 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.850050 2570 manager.go:349] Starting recovery of all containers Sep 16 18:45:12.854630 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.854600 2570 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 18:45:12.855135 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.855103 2570 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 18:45:12.855789 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.854685 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-175.ec2.internal.18d5e1736e7a7f9c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-175.ec2.internal,UID:ip-10-0-142-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-175.ec2.internal,},FirstTimestamp:2026-09-16 18:45:12.840650652 +0000 UTC m=+0.383956401,LastTimestamp:2026-09-16 18:45:12.840650652 +0000 UTC m=+0.383956401,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-175.ec2.internal,}" Sep 16 18:45:12.855895 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.855845 2570 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-175.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 18:45:12.864208 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.864190 2570 manager.go:354] Recovery completed Sep 16 18:45:12.868435 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.868422 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.869998 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.869978 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dt9jx" Sep 16 18:45:12.870073 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.870022 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.870073 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.870057 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.870073 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.870069 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.870675 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.870660 2570 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 18:45:12.870675 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.870674 2570 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 18:45:12.870798 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.870674 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.870798 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.870694 2570 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:45:12.871387 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.871372 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.871441 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.871397 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.871441 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.871405 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.872902 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.872886 2570 policy_none.go:49] "None policy: Start" Sep 16 18:45:12.872902 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.872763 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-175.ec2.internal.18d5e173703aed6c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-175.ec2.internal,UID:ip-10-0-142-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-175.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-175.ec2.internal,},FirstTimestamp:2026-09-16 18:45:12.870038892 +0000 UTC m=+0.413344641,LastTimestamp:2026-09-16 18:45:12.870038892 +0000 UTC m=+0.413344641,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-175.ec2.internal,}" Sep 16 18:45:12.872902 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.872903 2570 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 18:45:12.873033 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.872915 2570 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 18:45:12.874828 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.874816 2570 policy_none.go:47] "Start" Sep 16 18:45:12.881069 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.881053 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dt9jx" Sep 16 18:45:12.883268 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.883199 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-175.ec2.internal.18d5e173703b4a24 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-175.ec2.internal,UID:ip-10-0-142-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-175.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-175.ec2.internal,},FirstTimestamp:2026-09-16 18:45:12.870062628 +0000 UTC m=+0.413368378,LastTimestamp:2026-09-16 18:45:12.870062628 +0000 UTC m=+0.413368378,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-175.ec2.internal,}" Sep 16 18:45:12.885145 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.885071 2570 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-175.ec2.internal.18d5e173703b715b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-175.ec2.internal,UID:ip-10-0-142-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-175.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-175.ec2.internal,},FirstTimestamp:2026-09-16 18:45:12.870072667 +0000 UTC m=+0.413378417,LastTimestamp:2026-09-16 18:45:12.870072667 +0000 UTC m=+0.413378417,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-175.ec2.internal,}" Sep 16 18:45:12.913615 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.913600 2570 manager.go:337] "Starting Device Plugin manager" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.913637 2570 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.913647 2570 server.go:87] "Starting device plugin registration server" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.913974 2570 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.913988 2570 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.914218 2570 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.914327 2570 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.914336 2570 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.914968 2570 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 18:45:12.933489 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.915022 2570 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:12.963137 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.963098 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 18:45:12.964636 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.964621 2570 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 18:45:12.964726 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.964680 2570 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 18:45:12.964726 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.964708 2570 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 18:45:12.964820 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:12.964757 2570 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 18:45:12.973346 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:12.973326 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:13.014391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.014372 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:13.015748 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.015730 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:13.015832 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.015763 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:13.015832 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.015791 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:13.015832 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.015820 2570 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.033699 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.033642 2570 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.033699 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.033669 2570 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-175.ec2.internal\": node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.065795 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.065757 2570 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal"] Sep 16 18:45:13.065922 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.065829 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:13.067350 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.067329 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:13.067444 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.067369 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:13.067444 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.067387 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:13.069585 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.069571 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:13.069730 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.069715 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.069825 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.069750 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:13.070337 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.070316 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:13.070425 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.070319 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:13.070425 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.070369 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:13.070425 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.070380 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:13.070425 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.070345 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:13.070425 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.070417 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:13.072600 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.072582 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.072670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.072612 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:13.073354 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.073325 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:13.073412 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.073364 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:13.073412 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.073375 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:13.078918 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.078901 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.093402 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.093380 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-175.ec2.internal\" not found" node="ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.097652 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.097636 2570 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-175.ec2.internal\" not found" node="ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.150741 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.150710 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76eb21753bebddd59a75841e3efb8eea-config\") pod \"kube-apiserver-proxy-ip-10-0-142-175.ec2.internal\" (UID: \"76eb21753bebddd59a75841e3efb8eea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.150741 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.150741 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d03e3ae928eb1c2bb401b363ca17a1e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal\" (UID: \"d03e3ae928eb1c2bb401b363ca17a1e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.150953 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.150759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d03e3ae928eb1c2bb401b363ca17a1e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal\" (UID: \"d03e3ae928eb1c2bb401b363ca17a1e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.179390 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.179367 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.251853 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.251824 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d03e3ae928eb1c2bb401b363ca17a1e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal\" (UID: \"d03e3ae928eb1c2bb401b363ca17a1e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.251971 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.251855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76eb21753bebddd59a75841e3efb8eea-config\") pod \"kube-apiserver-proxy-ip-10-0-142-175.ec2.internal\" (UID: \"76eb21753bebddd59a75841e3efb8eea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.251971 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.251892 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d03e3ae928eb1c2bb401b363ca17a1e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal\" (UID: \"d03e3ae928eb1c2bb401b363ca17a1e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.251971 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.251917 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d03e3ae928eb1c2bb401b363ca17a1e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal\" (UID: \"d03e3ae928eb1c2bb401b363ca17a1e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.251971 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.251913 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d03e3ae928eb1c2bb401b363ca17a1e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal\" (UID: \"d03e3ae928eb1c2bb401b363ca17a1e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.252097 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.251968 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/76eb21753bebddd59a75841e3efb8eea-config\") pod \"kube-apiserver-proxy-ip-10-0-142-175.ec2.internal\" (UID: \"76eb21753bebddd59a75841e3efb8eea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.279948 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.279922 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.380698 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.380630 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.395832 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.395808 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.401258 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.400506 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" Sep 16 18:45:13.481548 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.481370 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.581923 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.581894 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.682443 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.682370 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.783018 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.782989 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.791164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.791138 2570 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 18:45:13.791307 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.791290 2570 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 18:45:13.884016 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.883988 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:13.886154 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.886118 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:13 +0000 UTC" deadline="2028-05-05 17:19:39.903644344 +0000 UTC" Sep 16 18:45:13.886154 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.886145 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14326h34m26.017502596s" Sep 16 18:45:13.953652 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.953582 2570 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:13.955566 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.955544 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:13.955688 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.955582 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:13.955688 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:13.955608 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:13.984422 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:13.984394 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:14.058558 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:14.058519 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03e3ae928eb1c2bb401b363ca17a1e5.slice/crio-57c65fa68b4e1bfb0edf3f3d488b130663adcfefac0a6f7ea57ff2ed97cbc957 WatchSource:0}: Error finding container 57c65fa68b4e1bfb0edf3f3d488b130663adcfefac0a6f7ea57ff2ed97cbc957: Status 404 returned error can't find the container with id 57c65fa68b4e1bfb0edf3f3d488b130663adcfefac0a6f7ea57ff2ed97cbc957 Sep 16 18:45:14.059378 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:14.059360 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76eb21753bebddd59a75841e3efb8eea.slice/crio-4977fe854640ba58f6892c9a8d815d40d3b1d7972bce418d153c0c70bfab80fc WatchSource:0}: Error finding container 4977fe854640ba58f6892c9a8d815d40d3b1d7972bce418d153c0c70bfab80fc: Status 404 returned error can't find the container with id 4977fe854640ba58f6892c9a8d815d40d3b1d7972bce418d153c0c70bfab80fc Sep 16 18:45:14.064448 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.064435 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:45:14.085141 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:14.085120 2570 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-175.ec2.internal\" not found" Sep 16 18:45:14.141698 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.141670 2570 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:14.151020 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.150994 2570 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:14.248910 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.248861 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" Sep 16 18:45:14.262261 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.262234 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:45:14.263680 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.263666 2570 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" Sep 16 18:45:14.273079 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.273056 2570 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:45:14.328066 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.328038 2570 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:14.827514 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.827472 2570 apiserver.go:52] "Watching apiserver" Sep 16 18:45:14.837034 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.837003 2570 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 18:45:14.838160 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.838128 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-r4877","kube-system/konnectivity-agent-q78m7","openshift-cluster-node-tuning-operator/tuned-chb9l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal","openshift-multus/multus-78bv4","openshift-multus/network-metrics-daemon-mllt4","openshift-network-diagnostics/network-check-target-pxxb4","openshift-ovn-kubernetes/ovnkube-node-nz9xs","kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8","openshift-dns/node-resolver-94bzk","openshift-image-registry/node-ca-q26jb","openshift-multus/multus-additional-cni-plugins-4wszm","openshift-network-operator/iptables-alerter-jvkpp"] Sep 16 18:45:14.840853 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.840821 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.843482 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.843458 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.845684 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.845659 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.846502 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.846460 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.846672 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.846655 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 18:45:14.847093 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.846966 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8dqkm\"" Sep 16 18:45:14.847093 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.846977 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.848100 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.848079 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.848694 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.848673 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 18:45:14.848907 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.848887 2570 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 18:45:14.849202 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.849181 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 18:45:14.850843 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.850594 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.850843 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.850624 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.851338 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.850899 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 18:45:14.853638 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.852905 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.853638 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.853160 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:14.853638 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:14.853233 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:14.856286 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.856084 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.857323 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.857304 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-95nbs\"" Sep 16 18:45:14.857592 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.857576 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.857751 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.857736 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kpj5f\"" Sep 16 18:45:14.858003 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.857986 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.858185 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.858171 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-75hml\"" Sep 16 18:45:14.858671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.858389 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 18:45:14.858671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.858560 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 18:45:14.858796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.858740 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzwtt\"" Sep 16 18:45:14.858867 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.858841 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.858905 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.858892 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.859123 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.859108 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:14.859196 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:14.859174 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:14.860969 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.860843 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:14.862040 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.861861 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-hostroot\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862040 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.861914 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-multus-certs\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862040 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.861944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-registration-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.862040 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.861972 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-device-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.862040 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862005 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-sys-fs\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.862216 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862042 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.862216 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862070 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-cni-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862216 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862097 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2c5b\" (UniqueName: \"kubernetes.io/projected/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-kube-api-access-d2c5b\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.862216 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862125 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvb9\" (UniqueName: \"kubernetes.io/projected/9a925473-9403-4198-a8cf-2ac41b9e73d0-kube-api-access-9tvb9\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.862216 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862171 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-modprobe-d\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.862376 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862259 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-systemd\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.862376 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862288 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-lib-modules\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.862376 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862313 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-tuned\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.862376 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862338 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-k8s-cni-cncf-io\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862376 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862354 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-cni-bin\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862381 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5vw\" (UniqueName: \"kubernetes.io/projected/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-kube-api-access-qk5vw\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862397 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-tmp\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862414 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcct\" (UniqueName: \"kubernetes.io/projected/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-kube-api-access-4kcct\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862428 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-socket-dir-parent\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862444 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862493 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz46x\" (UniqueName: \"kubernetes.io/projected/adcf6d46-0036-4802-a3bb-59bb77d2a040-kube-api-access-tz46x\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862508 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-system-cni-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862521 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-daemon-config\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862541 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-etc-kubernetes\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.862575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862566 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-sys\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.863257 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862661 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-netns\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.863257 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862719 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-kubelet\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.863257 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.862756 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.863437 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysconfig\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.863437 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863423 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-cni-binary-copy\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.863701 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863459 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-socket-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.863765 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863695 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-system-cni-dir\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.863765 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863733 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-kubernetes\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.863870 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysctl-d\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.863870 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863806 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysctl-conf\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.863870 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863840 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-var-lib-kubelet\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.863870 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863865 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-host\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.864090 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863887 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 18:45:14.864090 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863898 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-conf-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.864090 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.863950 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.864316 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.864098 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.864316 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.864141 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-cnibin\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.864316 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.864177 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-run\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.864316 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.864279 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.864529 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.864216 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adcf6d46-0036-4802-a3bb-59bb77d2a040-iptables-alerter-script\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.864529 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.864408 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adcf6d46-0036-4802-a3bb-59bb77d2a040-host-slash\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.865168 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.865144 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.865525 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.865507 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 18:45:14.865614 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.865525 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 18:45:14.866235 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.866211 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pg8wv\"" Sep 16 18:45:14.866797 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.866772 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 18:45:14.866981 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.866964 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 18:45:14.867388 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.867372 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 18:45:14.867594 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.867555 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tjbx\"" Sep 16 18:45:14.867855 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.867835 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:14.867970 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.867948 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-cnibin\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.868043 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.868006 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-cni-multus\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.868043 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:14.867946 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:14.870446 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.870323 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:14.871641 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.871614 2570 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:45:14.873026 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.872496 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hdqkd\"" Sep 16 18:45:14.873026 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.872661 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.873026 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.872907 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.873236 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.873214 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:14.875952 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.875921 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qc8ch\"" Sep 16 18:45:14.876041 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.875975 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.876254 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.876233 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 18:45:14.876332 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.876255 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.897011 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.896983 2570 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lwcmh" Sep 16 18:45:14.907671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.907641 2570 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lwcmh" Sep 16 18:45:14.950844 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.950818 2570 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 18:45:14.968209 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968182 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-socket-dir-parent\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.968329 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968222 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.968329 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968246 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.968329 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968272 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz46x\" (UniqueName: \"kubernetes.io/projected/adcf6d46-0036-4802-a3bb-59bb77d2a040-kube-api-access-tz46x\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.968499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968329 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-run-netns\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.968499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968372 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-systemd\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.968499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968400 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3953984a-4756-4979-9d8d-1f594e70ebd3-tmp-dir\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:14.968499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968405 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-socket-dir-parent\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.968499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-daemon-config\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.968499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968461 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-etc-kubernetes\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.968499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968484 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-sys\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968509 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nksm\" (UniqueName: \"kubernetes.io/projected/3953984a-4756-4979-9d8d-1f594e70ebd3-kube-api-access-4nksm\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968539 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwvxz\" (UniqueName: \"kubernetes.io/projected/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-kube-api-access-mwvxz\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968555 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-etc-kubernetes\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968564 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-etc-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968589 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3953984a-4756-4979-9d8d-1f594e70ebd3-hosts-file\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968639 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-cni-binary-copy\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968658 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-sys\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968669 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-system-cni-dir\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968710 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-system-cni-dir\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968703 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysctl-conf\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-host\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968796 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968832 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-ovnkube-config\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968854 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd72ad2-4fcc-48c2-8bce-1790994113ad-host\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdd72ad2-4fcc-48c2-8bce-1790994113ad-serviceca\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:14.968918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968917 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968945 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysctl-conf\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968964 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-cni-binary-copy\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968969 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.968912 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-host\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969021 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969075 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-daemon-config\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969132 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-kubelet\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969166 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kkxt\" (UniqueName: \"kubernetes.io/projected/bdd72ad2-4fcc-48c2-8bce-1790994113ad-kube-api-access-5kkxt\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969191 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-multus-certs\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-registration-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969222 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-etc-selinux\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969233 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969212 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" event={"ID":"d03e3ae928eb1c2bb401b363ca17a1e5","Type":"ContainerStarted","Data":"57c65fa68b4e1bfb0edf3f3d488b130663adcfefac0a6f7ea57ff2ed97cbc957"} Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969251 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-run-ovn-kubernetes\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969298 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-multus-certs\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.969776 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969304 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29f12a00-96bd-4697-a24f-88f121e439b0-konnectivity-ca\") pod \"konnectivity-agent-q78m7\" (UID: \"29f12a00-96bd-4697-a24f-88f121e439b0\") " pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969301 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-registration-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969350 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-cni-binary-copy\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969355 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2c5b\" (UniqueName: \"kubernetes.io/projected/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-kube-api-access-d2c5b\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969384 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-tuned\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-ovnkube-script-lib\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969548 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969595 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-node-log\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969634 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-cni-netd\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969657 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969689 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-env-overrides\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969857 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-system-cni-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969858 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9a925473-9403-4198-a8cf-2ac41b9e73d0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969912 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74b974f7-2018-46ff-8bce-901fb9018019-dbus\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969864 2570 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969945 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-netns\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969925 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-system-cni-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969971 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-kubelet\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969977 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-netns\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.969995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970023 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysconfig\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970042 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-kubelet\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970048 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970125 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysconfig\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970155 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adcf6d46-0036-4802-a3bb-59bb77d2a040-iptables-alerter-script\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adcf6d46-0036-4802-a3bb-59bb77d2a040-host-slash\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-socket-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970252 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-kubernetes\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970257 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/adcf6d46-0036-4802-a3bb-59bb77d2a040-host-slash\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970292 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysctl-d\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-kubernetes\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970319 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-var-lib-kubelet\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.971515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970390 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-ovn\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970416 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29f12a00-96bd-4697-a24f-88f121e439b0-agent-certs\") pod \"konnectivity-agent-q78m7\" (UID: \"29f12a00-96bd-4697-a24f-88f121e439b0\") " pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970425 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-sysctl-d\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970440 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-conf-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970465 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-cnibin\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970487 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-run\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970485 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-var-lib-kubelet\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-systemd-units\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970547 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a925473-9403-4198-a8cf-2ac41b9e73d0-cnibin\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970534 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-socket-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970554 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21641444-a6be-4e57-8f10-7bf854e3ab77-ovn-node-metrics-cert\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970615 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-run\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970630 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkljf\" (UniqueName: \"kubernetes.io/projected/21641444-a6be-4e57-8f10-7bf854e3ab77-kube-api-access-kkljf\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970646 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-conf-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970691 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-cnibin\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970780 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/adcf6d46-0036-4802-a3bb-59bb77d2a040-iptables-alerter-script\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970810 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-cnibin\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970822 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-cni-multus\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.972256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970887 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-hostroot\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970925 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-device-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970951 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-sys-fs\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970973 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-modprobe-d\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970983 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-hostroot\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.970990 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-cni-multus\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971025 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-sys-fs\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971035 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-device-dir\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971053 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-systemd\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971106 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-systemd\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971105 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-modprobe-d\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971140 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-lib-modules\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971199 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-cni-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvb9\" (UniqueName: \"kubernetes.io/projected/9a925473-9403-4198-a8cf-2ac41b9e73d0-kube-api-access-9tvb9\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971253 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-tmp\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971280 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-slash\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971310 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74b974f7-2018-46ff-8bce-901fb9018019-kubelet-config\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:14.973099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971336 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-k8s-cni-cncf-io\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971367 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-cni-bin\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971393 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5vw\" (UniqueName: \"kubernetes.io/projected/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-kube-api-access-qk5vw\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcct\" (UniqueName: \"kubernetes.io/projected/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-kube-api-access-4kcct\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971464 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-var-lib-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971494 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-log-socket\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971510 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-multus-cni-dir\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971520 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-cni-bin\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971311 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-lib-modules\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971629 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-run-k8s-cni-cncf-io\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.971677 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-host-var-lib-cni-bin\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.973589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.972603 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" event={"ID":"76eb21753bebddd59a75841e3efb8eea","Type":"ContainerStarted","Data":"4977fe854640ba58f6892c9a8d815d40d3b1d7972bce418d153c0c70bfab80fc"} Sep 16 18:45:14.974805 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.974782 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-etc-tuned\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.974913 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.974854 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-tmp\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.986421 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.986393 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2c5b\" (UniqueName: \"kubernetes.io/projected/c7aa2dec-55c2-4f60-b5c1-e8a439817b9d-kube-api-access-d2c5b\") pod \"aws-ebs-csi-driver-node-vqhh8\" (UID: \"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:14.987610 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.987563 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz46x\" (UniqueName: \"kubernetes.io/projected/adcf6d46-0036-4802-a3bb-59bb77d2a040-kube-api-access-tz46x\") pod \"iptables-alerter-jvkpp\" (UID: \"adcf6d46-0036-4802-a3bb-59bb77d2a040\") " pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:14.988744 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.988723 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5vw\" (UniqueName: \"kubernetes.io/projected/dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5-kube-api-access-qk5vw\") pod \"multus-78bv4\" (UID: \"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5\") " pod="openshift-multus/multus-78bv4" Sep 16 18:45:14.988940 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.988870 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcct\" (UniqueName: \"kubernetes.io/projected/e84f714a-7c66-450d-8aa0-5f96d4cad5bb-kube-api-access-4kcct\") pod \"tuned-chb9l\" (UID: \"e84f714a-7c66-450d-8aa0-5f96d4cad5bb\") " pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:14.989344 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:14.989321 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvb9\" (UniqueName: \"kubernetes.io/projected/9a925473-9403-4198-a8cf-2ac41b9e73d0-kube-api-access-9tvb9\") pod \"multus-additional-cni-plugins-4wszm\" (UID: \"9a925473-9403-4198-a8cf-2ac41b9e73d0\") " pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:15.072222 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072179 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-env-overrides\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072222 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072223 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:15.072466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072247 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74b974f7-2018-46ff-8bce-901fb9018019-dbus\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:15.072466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072277 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:15.072466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072376 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74b974f7-2018-46ff-8bce-901fb9018019-dbus\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:15.072466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072399 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-ovn\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072466 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.072384 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:15.072466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29f12a00-96bd-4697-a24f-88f121e439b0-agent-certs\") pod \"konnectivity-agent-q78m7\" (UID: \"29f12a00-96bd-4697-a24f-88f121e439b0\") " pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:15.072466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072448 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-systemd-units\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.072385 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.072494 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret podName:74b974f7-2018-46ff-8bce-901fb9018019 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.572469125 +0000 UTC m=+3.115774865 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret") pod "global-pull-secret-syncer-r4877" (UID: "74b974f7-2018-46ff-8bce-901fb9018019") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072474 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-ovn\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.072563 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.572547664 +0000 UTC m=+3.115853403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072473 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21641444-a6be-4e57-8f10-7bf854e3ab77-ovn-node-metrics-cert\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072598 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkljf\" (UniqueName: \"kubernetes.io/projected/21641444-a6be-4e57-8f10-7bf854e3ab77-kube-api-access-kkljf\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-slash\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072645 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74b974f7-2018-46ff-8bce-901fb9018019-kubelet-config\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072665 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-var-lib-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072682 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-log-socket\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-cni-bin\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072725 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-run-netns\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072741 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-systemd\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.072796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072756 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3953984a-4756-4979-9d8d-1f594e70ebd3-tmp-dir\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072800 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-systemd-units\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072843 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nksm\" (UniqueName: \"kubernetes.io/projected/3953984a-4756-4979-9d8d-1f594e70ebd3-kube-api-access-4nksm\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072875 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwvxz\" (UniqueName: \"kubernetes.io/projected/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-kube-api-access-mwvxz\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-etc-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072918 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-var-lib-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072944 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3953984a-4756-4979-9d8d-1f594e70ebd3-hosts-file\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072990 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3953984a-4756-4979-9d8d-1f594e70ebd3-hosts-file\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.072994 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073028 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-ovnkube-config\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073032 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3953984a-4756-4979-9d8d-1f594e70ebd3-tmp-dir\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073056 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd72ad2-4fcc-48c2-8bce-1790994113ad-host\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073062 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-env-overrides\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073082 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-log-socket\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073244 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-etc-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073302 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-run-netns\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073306 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.073516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073351 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-cni-bin\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073378 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74b974f7-2018-46ff-8bce-901fb9018019-kubelet-config\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073387 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-slash\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073085 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdd72ad2-4fcc-48c2-8bce-1790994113ad-serviceca\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-systemd\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073419 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bdd72ad2-4fcc-48c2-8bce-1790994113ad-serviceca\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-kubelet\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073420 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bdd72ad2-4fcc-48c2-8bce-1790994113ad-host\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073492 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-kubelet\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073497 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kkxt\" (UniqueName: \"kubernetes.io/projected/bdd72ad2-4fcc-48c2-8bce-1790994113ad-kube-api-access-5kkxt\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073530 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-run-ovn-kubernetes\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073557 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29f12a00-96bd-4697-a24f-88f121e439b0-konnectivity-ca\") pod \"konnectivity-agent-q78m7\" (UID: \"29f12a00-96bd-4697-a24f-88f121e439b0\") " pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073610 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-run-ovn-kubernetes\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073670 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-ovnkube-script-lib\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073700 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-node-log\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073801 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-cni-netd\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073822 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-ovnkube-config\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.074372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.075119 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073870 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-run-openvswitch\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.075119 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073931 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-host-cni-netd\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.075119 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.073979 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/21641444-a6be-4e57-8f10-7bf854e3ab77-node-log\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.075119 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.074213 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29f12a00-96bd-4697-a24f-88f121e439b0-konnectivity-ca\") pod \"konnectivity-agent-q78m7\" (UID: \"29f12a00-96bd-4697-a24f-88f121e439b0\") " pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:15.075119 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.074344 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/21641444-a6be-4e57-8f10-7bf854e3ab77-ovnkube-script-lib\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.075904 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.075883 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/21641444-a6be-4e57-8f10-7bf854e3ab77-ovn-node-metrics-cert\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.076068 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.076048 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29f12a00-96bd-4697-a24f-88f121e439b0-agent-certs\") pod \"konnectivity-agent-q78m7\" (UID: \"29f12a00-96bd-4697-a24f-88f121e439b0\") " pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:15.081309 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.081250 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:15.081309 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.081282 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:15.081309 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.081298 2570 projected.go:196] Error preparing data for projected volume kube-api-access-78c6n for pod openshift-network-diagnostics/network-check-target-pxxb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:15.081539 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.081373 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n podName:c2232d20-f65c-48c0-8165-64844c147398 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.581353061 +0000 UTC m=+3.124658825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-78c6n" (UniqueName: "kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n") pod "network-check-target-pxxb4" (UID: "c2232d20-f65c-48c0-8165-64844c147398") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:15.083312 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.083287 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwvxz\" (UniqueName: \"kubernetes.io/projected/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-kube-api-access-mwvxz\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:15.084393 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.084353 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nksm\" (UniqueName: \"kubernetes.io/projected/3953984a-4756-4979-9d8d-1f594e70ebd3-kube-api-access-4nksm\") pod \"node-resolver-94bzk\" (UID: \"3953984a-4756-4979-9d8d-1f594e70ebd3\") " pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:15.084479 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.084414 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kkxt\" (UniqueName: \"kubernetes.io/projected/bdd72ad2-4fcc-48c2-8bce-1790994113ad-kube-api-access-5kkxt\") pod \"node-ca-q26jb\" (UID: \"bdd72ad2-4fcc-48c2-8bce-1790994113ad\") " pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:15.084675 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.084655 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkljf\" (UniqueName: \"kubernetes.io/projected/21641444-a6be-4e57-8f10-7bf854e3ab77-kube-api-access-kkljf\") pod \"ovnkube-node-nz9xs\" (UID: \"21641444-a6be-4e57-8f10-7bf854e3ab77\") " pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.154377 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.154335 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" Sep 16 18:45:15.166543 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.166511 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4wszm" Sep 16 18:45:15.177160 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.177124 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-chb9l" Sep 16 18:45:15.186495 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.186469 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jvkpp" Sep 16 18:45:15.194603 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.194578 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-78bv4" Sep 16 18:45:15.203949 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.203912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:15.211815 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.211794 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:15.220967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.220942 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-94bzk" Sep 16 18:45:15.226233 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.226204 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q26jb" Sep 16 18:45:15.331252 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.331209 2570 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:15.578883 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.578769 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:15.578883 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.578815 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:15.579099 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.578972 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:15.579099 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.578982 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:15.579099 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.579048 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret podName:74b974f7-2018-46ff-8bce-901fb9018019 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.579025985 +0000 UTC m=+4.122331740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret") pod "global-pull-secret-syncer-r4877" (UID: "74b974f7-2018-46ff-8bce-901fb9018019") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:15.579099 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.579069 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.579059897 +0000 UTC m=+4.122365636 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:15.679609 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.679570 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:15.679791 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.679728 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:15.679791 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.679750 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:15.679791 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.679760 2570 projected.go:196] Error preparing data for projected volume kube-api-access-78c6n for pod openshift-network-diagnostics/network-check-target-pxxb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:15.679938 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.679828 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n podName:c2232d20-f65c-48c0-8165-64844c147398 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.679814527 +0000 UTC m=+4.223120281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-78c6n" (UniqueName: "kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n") pod "network-check-target-pxxb4" (UID: "c2232d20-f65c-48c0-8165-64844c147398") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:15.911032 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.910985 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:15 +0000 UTC" deadline="2028-04-11 13:31:22.537803131 +0000 UTC" Sep 16 18:45:15.911032 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.911019 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13746h46m6.626788219s" Sep 16 18:45:15.965057 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.965015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:15.965262 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:15.965016 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:15.965262 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.965173 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:15.965386 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:15.965268 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:16.066553 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:16.066504 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadcf6d46_0036_4802_a3bb_59bb77d2a040.slice/crio-0c8a7cb38cca7ad071508d5aaebf4e814e94e1449b60aa9f02564f8340723c3e WatchSource:0}: Error finding container 0c8a7cb38cca7ad071508d5aaebf4e814e94e1449b60aa9f02564f8340723c3e: Status 404 returned error can't find the container with id 0c8a7cb38cca7ad071508d5aaebf4e814e94e1449b60aa9f02564f8340723c3e Sep 16 18:45:16.070859 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:16.070602 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd72ad2_4fcc_48c2_8bce_1790994113ad.slice/crio-95908949d31e8535d8a358edb41bc70864a4b9c95900086af9c2a41ef7329322 WatchSource:0}: Error finding container 95908949d31e8535d8a358edb41bc70864a4b9c95900086af9c2a41ef7329322: Status 404 returned error can't find the container with id 95908949d31e8535d8a358edb41bc70864a4b9c95900086af9c2a41ef7329322 Sep 16 18:45:16.077992 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:16.077961 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7aa2dec_55c2_4f60_b5c1_e8a439817b9d.slice/crio-db96dda5dd6106aeb755b523efe0daab727495ee53694435f3bb9f4f824eddfc WatchSource:0}: Error finding container db96dda5dd6106aeb755b523efe0daab727495ee53694435f3bb9f4f824eddfc: Status 404 returned error can't find the container with id db96dda5dd6106aeb755b523efe0daab727495ee53694435f3bb9f4f824eddfc Sep 16 18:45:16.078492 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:16.078467 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf8d8d6_7154_4f6e_95ef_b7894c9d5ab5.slice/crio-37c596aff2fa2a680df99c3b4dea8f74c31e1fd6d1d6f3785857fe79667f2093 WatchSource:0}: Error finding container 37c596aff2fa2a680df99c3b4dea8f74c31e1fd6d1d6f3785857fe79667f2093: Status 404 returned error can't find the container with id 37c596aff2fa2a680df99c3b4dea8f74c31e1fd6d1d6f3785857fe79667f2093 Sep 16 18:45:16.082081 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:16.082042 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3953984a_4756_4979_9d8d_1f594e70ebd3.slice/crio-14a1b296bf68a05cf6f219eced3d5ddaf35da1bdd565e60d6a60f18556dabf1b WatchSource:0}: Error finding container 14a1b296bf68a05cf6f219eced3d5ddaf35da1bdd565e60d6a60f18556dabf1b: Status 404 returned error can't find the container with id 14a1b296bf68a05cf6f219eced3d5ddaf35da1bdd565e60d6a60f18556dabf1b Sep 16 18:45:16.099300 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:16.099170 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84f714a_7c66_450d_8aa0_5f96d4cad5bb.slice/crio-8d69d60ae22be89307ec6d2ccc2e215e9e7faa1970e655461339b6ed1c563653 WatchSource:0}: Error finding container 8d69d60ae22be89307ec6d2ccc2e215e9e7faa1970e655461339b6ed1c563653: Status 404 returned error can't find the container with id 8d69d60ae22be89307ec6d2ccc2e215e9e7faa1970e655461339b6ed1c563653 Sep 16 18:45:16.588978 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:16.588655 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:16.589200 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:16.589002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:16.589200 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.589086 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:16.589200 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.589171 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:45:18.589148157 +0000 UTC m=+6.132453900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:16.589200 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.589187 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:16.589454 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.589242 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret podName:74b974f7-2018-46ff-8bce-901fb9018019 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:18.58922699 +0000 UTC m=+6.132532735 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret") pod "global-pull-secret-syncer-r4877" (UID: "74b974f7-2018-46ff-8bce-901fb9018019") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:16.689870 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:16.689764 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:16.690066 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.690010 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:16.690066 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.690032 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:16.690066 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.690044 2570 projected.go:196] Error preparing data for projected volume kube-api-access-78c6n for pod openshift-network-diagnostics/network-check-target-pxxb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:16.690250 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.690112 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n podName:c2232d20-f65c-48c0-8165-64844c147398 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:18.69009392 +0000 UTC m=+6.233399683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-78c6n" (UniqueName: "kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n") pod "network-check-target-pxxb4" (UID: "c2232d20-f65c-48c0-8165-64844c147398") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:16.912016 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:16.911857 2570 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:15 +0000 UTC" deadline="2028-05-13 09:10:13.415617967 +0000 UTC" Sep 16 18:45:16.912016 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:16.911948 2570 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14510h24m56.503675315s" Sep 16 18:45:16.966634 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:16.966083 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:16.966634 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:16.966242 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:17.002768 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.002701 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wszm" event={"ID":"9a925473-9403-4198-a8cf-2ac41b9e73d0","Type":"ContainerStarted","Data":"8fba36e45bbb7b83579b7273aa329bac6f3432307d9c8c40b25c74cf6d7fb691"} Sep 16 18:45:17.023135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.023087 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-78bv4" event={"ID":"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5","Type":"ContainerStarted","Data":"37c596aff2fa2a680df99c3b4dea8f74c31e1fd6d1d6f3785857fe79667f2093"} Sep 16 18:45:17.035697 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.035655 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q26jb" event={"ID":"bdd72ad2-4fcc-48c2-8bce-1790994113ad","Type":"ContainerStarted","Data":"95908949d31e8535d8a358edb41bc70864a4b9c95900086af9c2a41ef7329322"} Sep 16 18:45:17.045413 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.045339 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-chb9l" event={"ID":"e84f714a-7c66-450d-8aa0-5f96d4cad5bb","Type":"ContainerStarted","Data":"8d69d60ae22be89307ec6d2ccc2e215e9e7faa1970e655461339b6ed1c563653"} Sep 16 18:45:17.052796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.052684 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q78m7" event={"ID":"29f12a00-96bd-4697-a24f-88f121e439b0","Type":"ContainerStarted","Data":"33aa2367b2b96cb778d477a5a9f97a5c94676f8bf9df44ec58d48b9cd06d40dc"} Sep 16 18:45:17.066993 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.066950 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-94bzk" event={"ID":"3953984a-4756-4979-9d8d-1f594e70ebd3","Type":"ContainerStarted","Data":"14a1b296bf68a05cf6f219eced3d5ddaf35da1bdd565e60d6a60f18556dabf1b"} Sep 16 18:45:17.076060 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.076019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" event={"ID":"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d","Type":"ContainerStarted","Data":"db96dda5dd6106aeb755b523efe0daab727495ee53694435f3bb9f4f824eddfc"} Sep 16 18:45:17.090316 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.090200 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jvkpp" event={"ID":"adcf6d46-0036-4802-a3bb-59bb77d2a040","Type":"ContainerStarted","Data":"0c8a7cb38cca7ad071508d5aaebf4e814e94e1449b60aa9f02564f8340723c3e"} Sep 16 18:45:17.105064 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.105024 2570 generic.go:356] "Generic (PLEG): container finished" podID="d03e3ae928eb1c2bb401b363ca17a1e5" containerID="f35c4c2e4b26b82dc8c59d18f7af5a0ed1a241a10a0e15b85ffe48e378a010af" exitCode=0 Sep 16 18:45:17.105256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.105139 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" event={"ID":"d03e3ae928eb1c2bb401b363ca17a1e5","Type":"ContainerDied","Data":"f35c4c2e4b26b82dc8c59d18f7af5a0ed1a241a10a0e15b85ffe48e378a010af"} Sep 16 18:45:17.133535 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.133496 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" event={"ID":"76eb21753bebddd59a75841e3efb8eea","Type":"ContainerStarted","Data":"703c1767e52f1ae28f2b655558e4898521050d2805c617c456ccf2dc6afefea7"} Sep 16 18:45:17.145741 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.145700 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"3a40592e1ab300d1182241035ff5d2163c58584a05d8afc9f1a18c54c1e6414c"} Sep 16 18:45:17.965108 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.965015 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:17.965108 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:17.965059 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:17.965806 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:17.965276 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:17.965806 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:17.965709 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:18.159343 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:18.159304 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" event={"ID":"d03e3ae928eb1c2bb401b363ca17a1e5","Type":"ContainerStarted","Data":"0945f81bd2e239bfba1075523fdd562165df1a1e0cf6ada03dc99bb85579e038"} Sep 16 18:45:18.178765 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:18.178691 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-175.ec2.internal" podStartSLOduration=4.178669753 podStartE2EDuration="4.178669753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:17.160272305 +0000 UTC m=+4.703578068" watchObservedRunningTime="2026-09-16 18:45:18.178669753 +0000 UTC m=+5.721975517" Sep 16 18:45:18.612267 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:18.612226 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:18.612450 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:18.612290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:18.612537 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.612461 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:18.612537 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.612464 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:18.612537 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.612534 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret podName:74b974f7-2018-46ff-8bce-901fb9018019 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.612516383 +0000 UTC m=+10.155822125 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret") pod "global-pull-secret-syncer-r4877" (UID: "74b974f7-2018-46ff-8bce-901fb9018019") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:18.612717 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.612551 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.612543366 +0000 UTC m=+10.155849103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:18.713442 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:18.712758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:18.713442 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.712964 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:18.713442 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.712989 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:18.713442 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.713003 2570 projected.go:196] Error preparing data for projected volume kube-api-access-78c6n for pod openshift-network-diagnostics/network-check-target-pxxb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:18.713442 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.713069 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n podName:c2232d20-f65c-48c0-8165-64844c147398 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.713049389 +0000 UTC m=+10.256355129 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-78c6n" (UniqueName: "kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n") pod "network-check-target-pxxb4" (UID: "c2232d20-f65c-48c0-8165-64844c147398") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:18.966616 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:18.965965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:18.966616 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:18.966128 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:19.965837 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:19.965800 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:19.966069 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:19.965985 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:19.966140 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:19.966095 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:19.966140 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:19.965984 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:20.965535 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:20.964975 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:20.965535 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:20.965132 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:21.965675 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:21.965585 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:21.966169 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:21.965746 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:21.966260 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:21.966230 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:21.966367 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:21.966344 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:22.650047 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:22.650002 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:22.650220 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:22.650062 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:22.650220 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.650210 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:22.650343 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.650282 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret podName:74b974f7-2018-46ff-8bce-901fb9018019 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:30.650263877 +0000 UTC m=+18.193569614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret") pod "global-pull-secret-syncer-r4877" (UID: "74b974f7-2018-46ff-8bce-901fb9018019") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:22.650675 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.650655 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:22.650738 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.650714 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:45:30.650697724 +0000 UTC m=+18.194003465 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:22.750896 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:22.750743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:22.751064 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.750991 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:22.751064 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.751019 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:22.751064 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.751032 2570 projected.go:196] Error preparing data for projected volume kube-api-access-78c6n for pod openshift-network-diagnostics/network-check-target-pxxb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:22.751203 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.751098 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n podName:c2232d20-f65c-48c0-8165-64844c147398 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:30.751078358 +0000 UTC m=+18.294384111 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-78c6n" (UniqueName: "kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n") pod "network-check-target-pxxb4" (UID: "c2232d20-f65c-48c0-8165-64844c147398") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:22.968048 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:22.967971 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:22.968458 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:22.968108 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:23.965064 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:23.965027 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:23.965248 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:23.965163 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:23.965577 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:23.965540 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:23.965690 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:23.965645 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:24.968239 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:24.968204 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:24.968644 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:24.968377 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:25.965722 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:25.965688 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:25.965907 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:25.965737 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:25.965907 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:25.965820 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:25.966039 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:25.965984 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:26.965626 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:26.965589 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:26.966088 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:26.965707 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:27.965084 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:27.965041 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:27.965281 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:27.965206 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:27.965281 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:27.965265 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:27.965422 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:27.965391 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:28.965148 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:28.965111 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:28.965608 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:28.965237 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:29.965424 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:29.965380 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:29.965941 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:29.965390 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:29.965941 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:29.965501 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:29.965941 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:29.965625 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:30.714526 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:30.714488 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:30.714526 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:30.714538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:30.714792 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.714664 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:30.714792 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.714665 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:30.714792 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.714738 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret podName:74b974f7-2018-46ff-8bce-901fb9018019 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.714719273 +0000 UTC m=+34.258025031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret") pod "global-pull-secret-syncer-r4877" (UID: "74b974f7-2018-46ff-8bce-901fb9018019") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:30.714792 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.714757 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.714746778 +0000 UTC m=+34.258052516 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:30.815847 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:30.815807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:30.816039 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.816020 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:30.816081 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.816052 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:30.816081 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.816066 2570 projected.go:196] Error preparing data for projected volume kube-api-access-78c6n for pod openshift-network-diagnostics/network-check-target-pxxb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:30.816156 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.816137 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n podName:c2232d20-f65c-48c0-8165-64844c147398 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.816118147 +0000 UTC m=+34.359423886 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-78c6n" (UniqueName: "kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n") pod "network-check-target-pxxb4" (UID: "c2232d20-f65c-48c0-8165-64844c147398") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:30.969068 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:30.968987 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:30.969499 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:30.969102 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:31.965446 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:31.965414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:31.965650 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:31.965414 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:31.965650 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:31.965580 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:31.965729 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:31.965689 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:32.966924 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:32.966409 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:32.966924 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:32.966537 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:33.965065 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:33.965031 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:33.965217 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:33.965037 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:33.965217 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:33.965138 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:33.965294 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:33.965225 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:34.965845 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:34.965809 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:34.966344 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:34.965986 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:35.202649 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.202324 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:45:35.202972 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.202939 2570 generic.go:356] "Generic (PLEG): container finished" podID="21641444-a6be-4e57-8f10-7bf854e3ab77" containerID="96ff6d2c9237203ca4932604a34db2b8c839e34fe807e8af727448ee10297a2f" exitCode=1 Sep 16 18:45:35.203085 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.202990 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"19a6936d8af5d04c83f35660abc349facf4d156897226e61c569bfddc33f4d4e"} Sep 16 18:45:35.203142 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.203090 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"62421cf3173d06d42bb92499c3172bf2df12854643f46027fcdf31a2f6c40648"} Sep 16 18:45:35.203203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.203142 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"ffd6428167a90152100e1fa9004e216fc4d8347c3bb0d881e2d389a09d60e8c3"} Sep 16 18:45:35.203203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.203192 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerDied","Data":"96ff6d2c9237203ca4932604a34db2b8c839e34fe807e8af727448ee10297a2f"} Sep 16 18:45:35.203295 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.203244 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"b9865c201833b28112581ca34b71c49cd4174383bb05efab10d88af119b27092"} Sep 16 18:45:35.204662 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.204637 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a925473-9403-4198-a8cf-2ac41b9e73d0" containerID="0bceaf8c8dd241582b14206a95a0148e643e9928a6f007ee11318a7d512d034f" exitCode=0 Sep 16 18:45:35.204791 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.204708 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wszm" event={"ID":"9a925473-9403-4198-a8cf-2ac41b9e73d0","Type":"ContainerDied","Data":"0bceaf8c8dd241582b14206a95a0148e643e9928a6f007ee11318a7d512d034f"} Sep 16 18:45:35.206341 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.206315 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-78bv4" event={"ID":"dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5","Type":"ContainerStarted","Data":"146d53fdffcb7f0d88f4302d8b86722a43def7b31602edccda93f0f839d3e44f"} Sep 16 18:45:35.208096 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.208068 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q26jb" event={"ID":"bdd72ad2-4fcc-48c2-8bce-1790994113ad","Type":"ContainerStarted","Data":"5062fd9ed8b85d44630d9ba8858b5984e6e55d96831068120c26f7df57c165df"} Sep 16 18:45:35.209524 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.209500 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-chb9l" event={"ID":"e84f714a-7c66-450d-8aa0-5f96d4cad5bb","Type":"ContainerStarted","Data":"a6150d7c11cd3f29b9bcc474086998d6a992351ef04b1865c72f29aeb5d5c171"} Sep 16 18:45:35.211025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.210992 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-q78m7" event={"ID":"29f12a00-96bd-4697-a24f-88f121e439b0","Type":"ContainerStarted","Data":"66089f152dde3ae5ad2d035168ca3375d644d242b23e342d8cacbcd7de52e64d"} Sep 16 18:45:35.212416 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.212396 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-94bzk" event={"ID":"3953984a-4756-4979-9d8d-1f594e70ebd3","Type":"ContainerStarted","Data":"4709a32ef501ad5e0d79d64fb1985359edfc5a7a1c65089bc74125973efdf2a8"} Sep 16 18:45:35.217122 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.217100 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" event={"ID":"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d","Type":"ContainerStarted","Data":"8b387c3598bac2a4ec35564dbbf77d00126d248143f1d65191849f56970b7c55"} Sep 16 18:45:35.228766 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.228708 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-175.ec2.internal" podStartSLOduration=21.228689908 podStartE2EDuration="21.228689908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:18.179550969 +0000 UTC m=+5.722856733" watchObservedRunningTime="2026-09-16 18:45:35.228689908 +0000 UTC m=+22.771995669" Sep 16 18:45:35.242768 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.242715 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-94bzk" podStartSLOduration=3.804481691 podStartE2EDuration="22.242699391s" totalImagesPullingTime="18.4382177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.242323141 +0000 UTC m=+22.785628900" watchObservedRunningTime="2026-09-16 18:45:35.242699391 +0000 UTC m=+22.786005150" Sep 16 18:45:35.258293 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.258242 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q26jb" podStartSLOduration=3.793354991 podStartE2EDuration="22.25822878s" totalImagesPullingTime="18.464873789s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.257837855 +0000 UTC m=+22.801143613" watchObservedRunningTime="2026-09-16 18:45:35.25822878 +0000 UTC m=+22.801534538" Sep 16 18:45:35.274312 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.274245 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-chb9l" podStartSLOduration=3.7002950119999998 podStartE2EDuration="22.27422759s" totalImagesPullingTime="18.573932578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.274145805 +0000 UTC m=+22.817451563" watchObservedRunningTime="2026-09-16 18:45:35.27422759 +0000 UTC m=+22.817533340" Sep 16 18:45:35.325117 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.325062 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-q78m7" podStartSLOduration=11.521307345 podStartE2EDuration="22.325045074s" totalImagesPullingTime="10.803737729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.324411268 +0000 UTC m=+22.867717028" watchObservedRunningTime="2026-09-16 18:45:35.325045074 +0000 UTC m=+22.868350832" Sep 16 18:45:35.374156 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.374094 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-78bv4" podStartSLOduration=3.873868623 podStartE2EDuration="22.374071402s" totalImagesPullingTime="18.500202779s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.372682747 +0000 UTC m=+22.915988507" watchObservedRunningTime="2026-09-16 18:45:35.374071402 +0000 UTC m=+22.917377162" Sep 16 18:45:35.965898 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.965851 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:35.965898 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:35.965882 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:35.966692 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:35.965992 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:35.966692 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:35.966080 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:36.094794 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.094755 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:36.095433 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.095414 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:36.116658 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.116632 2570 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 18:45:36.222223 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.222192 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" event={"ID":"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d","Type":"ContainerStarted","Data":"657194081d18b54e9830ee3bd9a619a620e3d49c15464d0b24505b15c03c1e3f"} Sep 16 18:45:36.223823 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.223797 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jvkpp" event={"ID":"adcf6d46-0036-4802-a3bb-59bb77d2a040","Type":"ContainerStarted","Data":"f600eccafe65f0676bece263a3b3dbb5b20d5e52e7eaad3896dc6a9bdbb59720"} Sep 16 18:45:36.227164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.227147 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:45:36.227745 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.227716 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"97038aaa5e636add3e622ae005171484a9236b21072e09435a1b7c1d8e51af70"} Sep 16 18:45:36.228337 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.228157 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:36.228813 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.228796 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-q78m7" Sep 16 18:45:36.242170 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.242131 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jvkpp" podStartSLOduration=4.772769133 podStartE2EDuration="23.242116079s" totalImagesPullingTime="18.469346946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:36.241576071 +0000 UTC m=+23.784881830" watchObservedRunningTime="2026-09-16 18:45:36.242116079 +0000 UTC m=+23.785421838" Sep 16 18:45:36.926053 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.925956 2570 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:36.116650504Z","UUID":"fbb6bb56-2781-4d14-845c-328d49589d3a","Handler":null,"Name":"","Endpoint":""} Sep 16 18:45:36.927616 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.927584 2570 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 18:45:36.927719 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.927626 2570 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 18:45:36.965149 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:36.965122 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:36.965309 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:36.965254 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:37.232627 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:37.232588 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" event={"ID":"c7aa2dec-55c2-4f60-b5c1-e8a439817b9d","Type":"ContainerStarted","Data":"d6026a9c03cf555421ad465d8428dddff7ea6e4acee319055f482f0fc75bbb5b"} Sep 16 18:45:37.253814 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:37.253765 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vqhh8" podStartSLOduration=3.8733547059999998 podStartE2EDuration="24.253749879s" totalImagesPullingTime="20.380395173s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:37.253138921 +0000 UTC m=+24.796444684" watchObservedRunningTime="2026-09-16 18:45:37.253749879 +0000 UTC m=+24.797055638" Sep 16 18:45:37.965451 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:37.965415 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:37.965647 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:37.965455 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:37.965647 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:37.965565 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:37.965759 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:37.965695 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:38.238604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:38.238523 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:45:38.239105 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:38.239013 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"fa633a7aea75915b82c5c38229350db3074148e9bbbdafe8718bdc357f730de7"} Sep 16 18:45:38.965943 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:38.965897 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:38.966104 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:38.966049 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:39.965283 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:39.965051 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:39.966161 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:39.965058 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:39.966161 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:39.965382 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:39.966161 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:39.965446 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:40.247913 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.247815 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:45:40.248287 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.248261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"d28556981f9d9547caa7298e4c4437cf11a8f53716b280ce8a001ce5fd787653"} Sep 16 18:45:40.248492 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.248472 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:40.248594 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.248542 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:40.248594 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.248590 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:40.248722 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.248702 2570 scope.go:117] "RemoveContainer" containerID="96ff6d2c9237203ca4932604a34db2b8c839e34fe807e8af727448ee10297a2f" Sep 16 18:45:40.250396 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.250376 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a925473-9403-4198-a8cf-2ac41b9e73d0" containerID="71b98319e9b0c52a2356f2e5bd51a187818831ac5077d83d0dd4c1fdca133758" exitCode=0 Sep 16 18:45:40.250482 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.250416 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wszm" event={"ID":"9a925473-9403-4198-a8cf-2ac41b9e73d0","Type":"ContainerDied","Data":"71b98319e9b0c52a2356f2e5bd51a187818831ac5077d83d0dd4c1fdca133758"} Sep 16 18:45:40.267202 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.267177 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:40.267439 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.267334 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:45:40.967199 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:40.967029 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:40.967538 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:40.967294 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:41.256388 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.256355 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:45:41.256734 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.256712 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" event={"ID":"21641444-a6be-4e57-8f10-7bf854e3ab77","Type":"ContainerStarted","Data":"126fcb887323ebbc6f38dbc55fdb1c290c4e13ad05bf2724af00f64502e68f2e"} Sep 16 18:45:41.258749 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.258724 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a925473-9403-4198-a8cf-2ac41b9e73d0" containerID="8e01c0d54f0dfd1ba124ec561d38ae9e3b62d193487b73b1f4de7bf1aa3958dc" exitCode=0 Sep 16 18:45:41.258884 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.258768 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wszm" event={"ID":"9a925473-9403-4198-a8cf-2ac41b9e73d0","Type":"ContainerDied","Data":"8e01c0d54f0dfd1ba124ec561d38ae9e3b62d193487b73b1f4de7bf1aa3958dc"} Sep 16 18:45:41.295428 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.295367 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" podStartSLOduration=9.847058565 podStartE2EDuration="28.295350407s" totalImagesPullingTime="18.448291842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:41.292416279 +0000 UTC m=+28.835722038" watchObservedRunningTime="2026-09-16 18:45:41.295350407 +0000 UTC m=+28.838656204" Sep 16 18:45:41.415476 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.415389 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r4877"] Sep 16 18:45:41.415616 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.415492 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:41.415616 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:41.415604 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:41.418612 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.418588 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mllt4"] Sep 16 18:45:41.418721 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.418704 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:41.418810 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:41.418794 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:41.422060 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.422036 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pxxb4"] Sep 16 18:45:41.422182 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:41.422128 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:41.422237 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:41.422218 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:42.965946 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:42.965905 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:42.966364 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:42.966021 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:42.966364 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:42.966098 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:42.966364 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:42.966014 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:42.966364 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:42.966125 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:42.966364 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:42.966222 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:43.265223 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:43.265136 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a925473-9403-4198-a8cf-2ac41b9e73d0" containerID="05626766d7628de50917e8a419cf23cbf7e1f7d0597b4a894d52eaba6c190b53" exitCode=0 Sep 16 18:45:43.265223 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:43.265174 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wszm" event={"ID":"9a925473-9403-4198-a8cf-2ac41b9e73d0","Type":"ContainerDied","Data":"05626766d7628de50917e8a419cf23cbf7e1f7d0597b4a894d52eaba6c190b53"} Sep 16 18:45:44.966205 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:44.965941 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:44.966642 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:44.965942 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:44.966642 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:44.966300 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:44.966642 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:44.965942 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:44.966642 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:44.966446 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:44.966642 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:44.966544 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:46.732277 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:46.732234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:46.732800 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:46.732290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:46.732800 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.732417 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:46.732800 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.732443 2570 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:46.732800 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.732496 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.732479732 +0000 UTC m=+66.275785490 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:46.732800 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.732518 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret podName:74b974f7-2018-46ff-8bce-901fb9018019 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.732507743 +0000 UTC m=+66.275813482 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret") pod "global-pull-secret-syncer-r4877" (UID: "74b974f7-2018-46ff-8bce-901fb9018019") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:46.833253 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:46.833214 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:46.833527 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.833367 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:46.833527 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.833389 2570 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:46.833527 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.833400 2570 projected.go:196] Error preparing data for projected volume kube-api-access-78c6n for pod openshift-network-diagnostics/network-check-target-pxxb4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:46.833527 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.833461 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n podName:c2232d20-f65c-48c0-8165-64844c147398 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.83344556 +0000 UTC m=+66.376751323 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-78c6n" (UniqueName: "kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n") pod "network-check-target-pxxb4" (UID: "c2232d20-f65c-48c0-8165-64844c147398") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:46.965240 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:46.965206 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:46.965466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:46.965208 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:46.965466 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.965339 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pxxb4" podUID="c2232d20-f65c-48c0-8165-64844c147398" Sep 16 18:45:46.965466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:46.965358 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:46.965466 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.965449 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:45:46.965677 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:46.965528 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-r4877" podUID="74b974f7-2018-46ff-8bce-901fb9018019" Sep 16 18:45:48.247641 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.247613 2570 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-175.ec2.internal" event="NodeReady" Sep 16 18:45:48.248147 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.247729 2570 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 18:45:48.280202 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.280167 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a925473-9403-4198-a8cf-2ac41b9e73d0" containerID="d2f1d9da782daad29a80377e925996afa560b85d2a174a238d957325b2c12b4f" exitCode=0 Sep 16 18:45:48.280365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.280225 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wszm" event={"ID":"9a925473-9403-4198-a8cf-2ac41b9e73d0","Type":"ContainerDied","Data":"d2f1d9da782daad29a80377e925996afa560b85d2a174a238d957325b2c12b4f"} Sep 16 18:45:48.295032 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.295008 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-pvb5t"] Sep 16 18:45:48.306685 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.306663 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-95rsc"] Sep 16 18:45:48.306819 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.306801 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.310521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.310362 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 18:45:48.310521 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.310373 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 18:45:48.311616 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.311588 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgt44\"" Sep 16 18:45:48.330080 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.330052 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvb5t"] Sep 16 18:45:48.330229 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.330100 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-95rsc"] Sep 16 18:45:48.330229 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.330178 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-95rsc" Sep 16 18:45:48.334480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.333312 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 18:45:48.334480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.333898 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 18:45:48.334480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.334099 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z45rg\"" Sep 16 18:45:48.337335 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.337315 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 18:45:48.442548 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.442518 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzf2j\" (UniqueName: \"kubernetes.io/projected/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-kube-api-access-kzf2j\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.442673 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.442636 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-tmp-dir\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.442733 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.442671 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-config-volume\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.442733 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.442699 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2lwl\" (UniqueName: \"kubernetes.io/projected/22ca6920-60cc-4b51-8e27-f3647ad138e5-kube-api-access-q2lwl\") pod \"ingress-canary-95rsc\" (UID: \"22ca6920-60cc-4b51-8e27-f3647ad138e5\") " pod="openshift-ingress-canary/ingress-canary-95rsc" Sep 16 18:45:48.442837 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.442739 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.442837 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.442795 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22ca6920-60cc-4b51-8e27-f3647ad138e5-cert\") pod \"ingress-canary-95rsc\" (UID: \"22ca6920-60cc-4b51-8e27-f3647ad138e5\") " pod="openshift-ingress-canary/ingress-canary-95rsc" Sep 16 18:45:48.543882 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.543826 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-config-volume\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.544110 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.543889 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2lwl\" (UniqueName: \"kubernetes.io/projected/22ca6920-60cc-4b51-8e27-f3647ad138e5-kube-api-access-q2lwl\") pod \"ingress-canary-95rsc\" (UID: \"22ca6920-60cc-4b51-8e27-f3647ad138e5\") " pod="openshift-ingress-canary/ingress-canary-95rsc" Sep 16 18:45:48.544110 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.543912 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.544110 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.543937 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22ca6920-60cc-4b51-8e27-f3647ad138e5-cert\") pod \"ingress-canary-95rsc\" (UID: \"22ca6920-60cc-4b51-8e27-f3647ad138e5\") " pod="openshift-ingress-canary/ingress-canary-95rsc" Sep 16 18:45:48.544110 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.543958 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzf2j\" (UniqueName: \"kubernetes.io/projected/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-kube-api-access-kzf2j\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.544110 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.543993 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-tmp-dir\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.544110 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:48.544089 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:48.544395 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:48.544169 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls podName:fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.04414913 +0000 UTC m=+36.587454867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls") pod "dns-default-pvb5t" (UID: "fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e") : secret "dns-default-metrics-tls" not found Sep 16 18:45:48.544395 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.544297 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-tmp-dir\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.544480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.544458 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-config-volume\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.548206 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.548176 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/22ca6920-60cc-4b51-8e27-f3647ad138e5-cert\") pod \"ingress-canary-95rsc\" (UID: \"22ca6920-60cc-4b51-8e27-f3647ad138e5\") " pod="openshift-ingress-canary/ingress-canary-95rsc" Sep 16 18:45:48.554929 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.554908 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2lwl\" (UniqueName: \"kubernetes.io/projected/22ca6920-60cc-4b51-8e27-f3647ad138e5-kube-api-access-q2lwl\") pod \"ingress-canary-95rsc\" (UID: \"22ca6920-60cc-4b51-8e27-f3647ad138e5\") " pod="openshift-ingress-canary/ingress-canary-95rsc" Sep 16 18:45:48.556524 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.556506 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzf2j\" (UniqueName: \"kubernetes.io/projected/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-kube-api-access-kzf2j\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:48.642718 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.642685 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-95rsc" Sep 16 18:45:48.968758 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.968571 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:45:48.968937 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.968633 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:45:48.969024 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.968638 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:45:48.974295 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.974272 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7n2q\"" Sep 16 18:45:48.975029 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.974857 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmvxr\"" Sep 16 18:45:48.975029 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.974909 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:45:48.975029 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.974974 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:45:48.975029 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.975006 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:45:48.975294 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.975009 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:45:48.976497 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:48.976435 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-95rsc"] Sep 16 18:45:48.981565 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:45:48.981539 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22ca6920_60cc_4b51_8e27_f3647ad138e5.slice/crio-903803f4e706556e07e2a9d743174adb4093bc73926163a944090b06d6184512 WatchSource:0}: Error finding container 903803f4e706556e07e2a9d743174adb4093bc73926163a944090b06d6184512: Status 404 returned error can't find the container with id 903803f4e706556e07e2a9d743174adb4093bc73926163a944090b06d6184512 Sep 16 18:45:49.047960 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:49.047922 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:49.048128 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:49.048080 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:49.048170 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:49.048145 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls podName:fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.048129943 +0000 UTC m=+37.591435703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls") pod "dns-default-pvb5t" (UID: "fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e") : secret "dns-default-metrics-tls" not found Sep 16 18:45:49.285710 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:49.285623 2570 generic.go:356] "Generic (PLEG): container finished" podID="9a925473-9403-4198-a8cf-2ac41b9e73d0" containerID="f34ba8c1c2e2c6a3e579db1ce5fefffe0e37af4ec3d3507aa81750d6e0862a14" exitCode=0 Sep 16 18:45:49.286508 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:49.285709 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wszm" event={"ID":"9a925473-9403-4198-a8cf-2ac41b9e73d0","Type":"ContainerDied","Data":"f34ba8c1c2e2c6a3e579db1ce5fefffe0e37af4ec3d3507aa81750d6e0862a14"} Sep 16 18:45:49.286949 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:49.286920 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-95rsc" event={"ID":"22ca6920-60cc-4b51-8e27-f3647ad138e5","Type":"ContainerStarted","Data":"903803f4e706556e07e2a9d743174adb4093bc73926163a944090b06d6184512"} Sep 16 18:45:50.055340 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:50.055297 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:50.055530 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:50.055471 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:50.055596 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:50.055546 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls podName:fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.05552992 +0000 UTC m=+39.598835656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls") pod "dns-default-pvb5t" (UID: "fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e") : secret "dns-default-metrics-tls" not found Sep 16 18:45:50.295313 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:50.295279 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4wszm" event={"ID":"9a925473-9403-4198-a8cf-2ac41b9e73d0","Type":"ContainerStarted","Data":"c64b35e794160a473c8f8b0ca73b2ab34adb67c23757dafbee5cbcd883c40ff1"} Sep 16 18:45:50.319932 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:50.319567 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4wszm" podStartSLOduration=8.613906440000001 podStartE2EDuration="37.319544074s" totalImagesPullingTime="28.705637634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:50.318896249 +0000 UTC m=+37.862202002" watchObservedRunningTime="2026-09-16 18:45:50.319544074 +0000 UTC m=+37.862849834" Sep 16 18:45:52.074477 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:52.074281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:52.074477 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:52.074431 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:52.074957 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:52.074532 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls podName:fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.074516561 +0000 UTC m=+43.617822310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls") pod "dns-default-pvb5t" (UID: "fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e") : secret "dns-default-metrics-tls" not found Sep 16 18:45:52.301338 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:52.301307 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-95rsc" event={"ID":"22ca6920-60cc-4b51-8e27-f3647ad138e5","Type":"ContainerStarted","Data":"48160985c1e53f2a1432850728cf6a4d7aa78d58b0cca1d51486039c71719de7"} Sep 16 18:45:56.102440 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:45:56.102394 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:45:56.102819 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:56.102541 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:56.102819 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:45:56.102599 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls podName:fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.102585008 +0000 UTC m=+51.645890758 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls") pod "dns-default-pvb5t" (UID: "fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e") : secret "dns-default-metrics-tls" not found Sep 16 18:46:04.158159 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:04.158116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:46:04.158645 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:46:04.158233 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:04.158645 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:46:04.158288 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls podName:fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.158274837 +0000 UTC m=+67.701580573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls") pod "dns-default-pvb5t" (UID: "fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e") : secret "dns-default-metrics-tls" not found Sep 16 18:46:12.272035 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:12.272006 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-nz9xs" Sep 16 18:46:12.306207 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:12.306165 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-95rsc" podStartSLOduration=21.885346902 podStartE2EDuration="24.306150826s" totalImagesPullingTime="2.420803924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:52.323017422 +0000 UTC m=+39.866323181" watchObservedRunningTime="2026-09-16 18:46:12.306150826 +0000 UTC m=+59.849456584" Sep 16 18:46:18.758622 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.758581 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:46:18.758622 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.758626 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:46:18.761230 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.761209 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:46:18.761450 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.761432 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:46:18.769265 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:46:18.769239 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:46:18.769340 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:46:18.769303 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:47:22.769287031 +0000 UTC m=+130.312592780 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : secret "metrics-daemon-secret" not found Sep 16 18:46:18.772933 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.772912 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74b974f7-2018-46ff-8bce-901fb9018019-original-pull-secret\") pod \"global-pull-secret-syncer-r4877\" (UID: \"74b974f7-2018-46ff-8bce-901fb9018019\") " pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:46:18.859496 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.859461 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:46:18.862065 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.862044 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:46:18.872401 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.872381 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:46:18.882783 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.882759 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78c6n\" (UniqueName: \"kubernetes.io/projected/c2232d20-f65c-48c0-8165-64844c147398-kube-api-access-78c6n\") pod \"network-check-target-pxxb4\" (UID: \"c2232d20-f65c-48c0-8165-64844c147398\") " pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:46:18.980369 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.980334 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-r4877" Sep 16 18:46:18.993213 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:18.993187 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmvxr\"" Sep 16 18:46:19.001207 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:19.001184 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:46:19.282315 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:19.282180 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-r4877"] Sep 16 18:46:19.285803 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:46:19.285776 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b974f7_2018_46ff_8bce_901fb9018019.slice/crio-70b52d6d60266a6242bbf1b1753163c4aac29464aa6b5d42946c76b5c2561c7e WatchSource:0}: Error finding container 70b52d6d60266a6242bbf1b1753163c4aac29464aa6b5d42946c76b5c2561c7e: Status 404 returned error can't find the container with id 70b52d6d60266a6242bbf1b1753163c4aac29464aa6b5d42946c76b5c2561c7e Sep 16 18:46:19.317228 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:19.317198 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pxxb4"] Sep 16 18:46:19.320840 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:46:19.320816 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2232d20_f65c_48c0_8165_64844c147398.slice/crio-f392a058cc5f9e03ccca6d2edf9a9cc4dd0e03f338664ca062c3c7f0f8e7e515 WatchSource:0}: Error finding container f392a058cc5f9e03ccca6d2edf9a9cc4dd0e03f338664ca062c3c7f0f8e7e515: Status 404 returned error can't find the container with id f392a058cc5f9e03ccca6d2edf9a9cc4dd0e03f338664ca062c3c7f0f8e7e515 Sep 16 18:46:19.367611 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:19.367578 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r4877" event={"ID":"74b974f7-2018-46ff-8bce-901fb9018019","Type":"ContainerStarted","Data":"70b52d6d60266a6242bbf1b1753163c4aac29464aa6b5d42946c76b5c2561c7e"} Sep 16 18:46:19.368750 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:19.368724 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pxxb4" event={"ID":"c2232d20-f65c-48c0-8165-64844c147398","Type":"ContainerStarted","Data":"f392a058cc5f9e03ccca6d2edf9a9cc4dd0e03f338664ca062c3c7f0f8e7e515"} Sep 16 18:46:20.169421 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:20.169257 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:46:20.170036 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:46:20.169437 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:20.170036 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:46:20.169555 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls podName:fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:52.169534251 +0000 UTC m=+99.712840011 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls") pod "dns-default-pvb5t" (UID: "fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e") : secret "dns-default-metrics-tls" not found Sep 16 18:46:24.384809 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:24.384771 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-r4877" event={"ID":"74b974f7-2018-46ff-8bce-901fb9018019","Type":"ContainerStarted","Data":"fb8ab9dbf65e15b67bb0a8839ddb5a2d3a814d54dafdc94f7a6c0557ecf7995f"} Sep 16 18:46:24.386286 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:24.386262 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pxxb4" event={"ID":"c2232d20-f65c-48c0-8165-64844c147398","Type":"ContainerStarted","Data":"0e4fb8313a5c65326e1dc4ec287edee7f9b2db6cb7a1d15a238482beda851110"} Sep 16 18:46:24.386406 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:24.386390 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:46:24.403704 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:24.403655 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-r4877" podStartSLOduration=66.897945146 podStartE2EDuration="1m11.403639686s" totalImagesPullingTime="4.50569454s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:24.403186462 +0000 UTC m=+71.946492220" watchObservedRunningTime="2026-09-16 18:46:24.403639686 +0000 UTC m=+71.946945444" Sep 16 18:46:24.419527 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:24.419482 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pxxb4" podStartSLOduration=66.952194287 podStartE2EDuration="1m11.419470551s" totalImagesPullingTime="4.467276264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:24.419395538 +0000 UTC m=+71.962701298" watchObservedRunningTime="2026-09-16 18:46:24.419470551 +0000 UTC m=+71.962776302" Sep 16 18:46:52.201320 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:52.201279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:46:52.201800 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:46:52.201392 2570 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:52.201800 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:46:52.201451 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls podName:fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e nodeName:}" failed. No retries permitted until 2026-09-16 18:47:56.201435339 +0000 UTC m=+163.744741076 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls") pod "dns-default-pvb5t" (UID: "fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e") : secret "dns-default-metrics-tls" not found Sep 16 18:46:55.392095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:46:55.392067 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pxxb4" Sep 16 18:47:22.800839 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:22.800798 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:47:22.801343 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:22.800960 2570 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:47:22.801343 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:22.801024 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs podName:f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba nodeName:}" failed. No retries permitted until 2026-09-16 18:49:24.801009594 +0000 UTC m=+252.344315346 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs") pod "network-metrics-daemon-mllt4" (UID: "f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba") : secret "metrics-daemon-secret" not found Sep 16 18:47:32.956480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.956442 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4"] Sep 16 18:47:32.958280 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.958265 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4" Sep 16 18:47:32.961888 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.961851 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 16 18:47:32.962020 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.961906 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:32.962169 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.962154 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kp7f9\"" Sep 16 18:47:32.963581 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.963562 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8"] Sep 16 18:47:32.965395 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.965374 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:32.968364 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.968347 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 16 18:47:32.968478 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.968397 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kkldc\"" Sep 16 18:47:32.968842 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.968827 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 16 18:47:32.971254 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.971234 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4"] Sep 16 18:47:32.973619 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.973602 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:32.998434 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:32.998410 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8"] Sep 16 18:47:33.064738 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.064704 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs2hz\" (UniqueName: \"kubernetes.io/projected/bf8a0586-521c-431c-97f2-6d8faf653e38-kube-api-access-qs2hz\") pod \"volume-data-source-validator-86695b7cc7-s8zb4\" (UID: \"bf8a0586-521c-431c-97f2-6d8faf653e38\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4" Sep 16 18:47:33.064936 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.064754 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:33.064936 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.064835 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jplpb\" (UniqueName: \"kubernetes.io/projected/c017308a-5c89-4157-954c-2bc7f4456b12-kube-api-access-jplpb\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:33.072894 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.072826 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk"] Sep 16 18:47:33.075025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.075007 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pprg4"] Sep 16 18:47:33.075148 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.075131 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.076677 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.076661 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pprg4" Sep 16 18:47:33.079453 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.079417 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 16 18:47:33.079720 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.079703 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 16 18:47:33.079806 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.079784 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hhfck\"" Sep 16 18:47:33.080230 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.080211 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:33.080326 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.080276 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jv65c\"" Sep 16 18:47:33.081480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.081462 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 16 18:47:33.089577 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.089560 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pprg4"] Sep 16 18:47:33.090171 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.090155 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk"] Sep 16 18:47:33.165412 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.165378 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:33.165412 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.165421 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9mg\" (UniqueName: \"kubernetes.io/projected/26fa1a05-58b1-4135-9703-6e7775f03691-kube-api-access-2w9mg\") pod \"network-check-source-6f8b768455-pprg4\" (UID: \"26fa1a05-58b1-4135-9703-6e7775f03691\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pprg4" Sep 16 18:47:33.165671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.165477 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jplpb\" (UniqueName: \"kubernetes.io/projected/c017308a-5c89-4157-954c-2bc7f4456b12-kube-api-access-jplpb\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:33.165671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.165515 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-config\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.165671 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.165532 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:33.165671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.165589 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-serving-cert\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.165671 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.165600 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls podName:c017308a-5c89-4157-954c-2bc7f4456b12 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:33.665582036 +0000 UTC m=+141.208887779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-77gx8" (UID: "c017308a-5c89-4157-954c-2bc7f4456b12") : secret "samples-operator-tls" not found Sep 16 18:47:33.165671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.165628 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mggfw\" (UniqueName: \"kubernetes.io/projected/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-kube-api-access-mggfw\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.165671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.165661 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs2hz\" (UniqueName: \"kubernetes.io/projected/bf8a0586-521c-431c-97f2-6d8faf653e38-kube-api-access-qs2hz\") pod \"volume-data-source-validator-86695b7cc7-s8zb4\" (UID: \"bf8a0586-521c-431c-97f2-6d8faf653e38\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4" Sep 16 18:47:33.187193 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.187161 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-dws7b"] Sep 16 18:47:33.189281 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.189266 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.195931 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.195910 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 16 18:47:33.196029 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.196001 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qb8tt\"" Sep 16 18:47:33.201759 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.201739 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 18:47:33.201898 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.201883 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 16 18:47:33.207650 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.207592 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs2hz\" (UniqueName: \"kubernetes.io/projected/bf8a0586-521c-431c-97f2-6d8faf653e38-kube-api-access-qs2hz\") pod \"volume-data-source-validator-86695b7cc7-s8zb4\" (UID: \"bf8a0586-521c-431c-97f2-6d8faf653e38\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4" Sep 16 18:47:33.211733 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.211549 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 18:47:33.231301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.230754 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jplpb\" (UniqueName: \"kubernetes.io/projected/c017308a-5c89-4157-954c-2bc7f4456b12-kube-api-access-jplpb\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:33.231445 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.231301 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-dws7b"] Sep 16 18:47:33.235499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.235475 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 16 18:47:33.266795 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.266763 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16ac85c-afc9-484d-8513-fa3f16202d79-service-ca-bundle\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.266795 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.266795 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6ng\" (UniqueName: \"kubernetes.io/projected/d16ac85c-afc9-484d-8513-fa3f16202d79-kube-api-access-9h6ng\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.267030 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.266824 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d16ac85c-afc9-484d-8513-fa3f16202d79-serving-cert\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.267030 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.266849 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d16ac85c-afc9-484d-8513-fa3f16202d79-snapshots\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.267030 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.266934 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4" Sep 16 18:47:33.267137 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.267045 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-config\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.267180 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.267133 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-serving-cert\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.267180 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.267157 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mggfw\" (UniqueName: \"kubernetes.io/projected/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-kube-api-access-mggfw\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.267277 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.267212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9mg\" (UniqueName: \"kubernetes.io/projected/26fa1a05-58b1-4135-9703-6e7775f03691-kube-api-access-2w9mg\") pod \"network-check-source-6f8b768455-pprg4\" (UID: \"26fa1a05-58b1-4135-9703-6e7775f03691\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pprg4" Sep 16 18:47:33.267277 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.267233 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16ac85c-afc9-484d-8513-fa3f16202d79-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.267277 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.267254 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16ac85c-afc9-484d-8513-fa3f16202d79-tmp\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.267546 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.267523 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-config\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.269575 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.269558 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-serving-cert\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.287878 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.287831 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9mg\" (UniqueName: \"kubernetes.io/projected/26fa1a05-58b1-4135-9703-6e7775f03691-kube-api-access-2w9mg\") pod \"network-check-source-6f8b768455-pprg4\" (UID: \"26fa1a05-58b1-4135-9703-6e7775f03691\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-pprg4" Sep 16 18:47:33.297007 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.296980 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64f8c79549-7cwhm"] Sep 16 18:47:33.299845 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.299824 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.303667 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.303034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mggfw\" (UniqueName: \"kubernetes.io/projected/a289a67b-4e5d-49cd-ab8f-0141ee4bb620-kube-api-access-mggfw\") pod \"service-ca-operator-cf7c776b7-2jzhk\" (UID: \"a289a67b-4e5d-49cd-ab8f-0141ee4bb620\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.304358 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.304189 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 18:47:33.304358 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.304222 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s9n6t\"" Sep 16 18:47:33.304616 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.304598 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 18:47:33.304698 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.304679 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 18:47:33.319036 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.319011 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 18:47:33.325034 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.324844 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64f8c79549-7cwhm"] Sep 16 18:47:33.367637 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367601 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16ac85c-afc9-484d-8513-fa3f16202d79-service-ca-bundle\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.367637 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367636 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6ng\" (UniqueName: \"kubernetes.io/projected/d16ac85c-afc9-484d-8513-fa3f16202d79-kube-api-access-9h6ng\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.367909 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367656 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.367909 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367678 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d16ac85c-afc9-484d-8513-fa3f16202d79-serving-cert\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.367909 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367694 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d16ac85c-afc9-484d-8513-fa3f16202d79-snapshots\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.367909 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367761 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-bound-sa-token\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.367909 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367809 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-certificates\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.367909 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367837 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-installation-pull-secrets\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.368203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-trusted-ca\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.368203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.367961 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9777e5e6-2eed-4efd-b9aa-27084ce8debb-ca-trust-extracted\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.368203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.368031 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16ac85c-afc9-484d-8513-fa3f16202d79-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.368203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.368063 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16ac85c-afc9-484d-8513-fa3f16202d79-tmp\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.368203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.368094 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-image-registry-private-configuration\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.368203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.368127 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkphs\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-kube-api-access-bkphs\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.368716 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.368692 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d16ac85c-afc9-484d-8513-fa3f16202d79-tmp\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.368808 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.368756 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d16ac85c-afc9-484d-8513-fa3f16202d79-snapshots\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.368808 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.368784 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16ac85c-afc9-484d-8513-fa3f16202d79-service-ca-bundle\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.369084 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.369064 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d16ac85c-afc9-484d-8513-fa3f16202d79-trusted-ca-bundle\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.370572 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.370551 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d16ac85c-afc9-484d-8513-fa3f16202d79-serving-cert\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.385061 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.385043 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" Sep 16 18:47:33.385432 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.385417 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6ng\" (UniqueName: \"kubernetes.io/projected/d16ac85c-afc9-484d-8513-fa3f16202d79-kube-api-access-9h6ng\") pod \"insights-operator-5844b856db-dws7b\" (UID: \"d16ac85c-afc9-484d-8513-fa3f16202d79\") " pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.389919 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.389902 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pprg4" Sep 16 18:47:33.469317 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.469289 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-image-registry-private-configuration\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.469458 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.469327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkphs\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-kube-api-access-bkphs\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.469458 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.469350 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.469458 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.469373 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-bound-sa-token\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.469458 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.469400 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-certificates\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.469458 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.469424 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-installation-pull-secrets\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.469458 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.469456 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:47:33.469772 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.469467 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-trusted-ca\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.469772 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.469505 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9777e5e6-2eed-4efd-b9aa-27084ce8debb-ca-trust-extracted\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.469772 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.469472 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f8c79549-7cwhm: secret "image-registry-tls" not found Sep 16 18:47:33.469772 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.469648 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls podName:9777e5e6-2eed-4efd-b9aa-27084ce8debb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:33.969627031 +0000 UTC m=+141.512932784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls") pod "image-registry-64f8c79549-7cwhm" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb") : secret "image-registry-tls" not found Sep 16 18:47:33.470213 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.470191 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9777e5e6-2eed-4efd-b9aa-27084ce8debb-ca-trust-extracted\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.470488 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.470289 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-certificates\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.470614 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.470596 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-trusted-ca\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.472333 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.472313 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-image-registry-private-configuration\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.472450 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.472364 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-installation-pull-secrets\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.478633 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.478607 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-bound-sa-token\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.479135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.479119 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkphs\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-kube-api-access-bkphs\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.497372 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.497355 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-dws7b" Sep 16 18:47:33.557053 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.557018 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4"] Sep 16 18:47:33.560044 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:47:33.559991 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8a0586_521c_431c_97f2_6d8faf653e38.slice/crio-4dd6761dce24662071b370d3ea464a60ea4b8ac0bb04d7fde1d657eafe178d4a WatchSource:0}: Error finding container 4dd6761dce24662071b370d3ea464a60ea4b8ac0bb04d7fde1d657eafe178d4a: Status 404 returned error can't find the container with id 4dd6761dce24662071b370d3ea464a60ea4b8ac0bb04d7fde1d657eafe178d4a Sep 16 18:47:33.671041 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.670991 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:33.671196 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.671137 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:33.671242 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.671201 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls podName:c017308a-5c89-4157-954c-2bc7f4456b12 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.671186626 +0000 UTC m=+142.214492363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-77gx8" (UID: "c017308a-5c89-4157-954c-2bc7f4456b12") : secret "samples-operator-tls" not found Sep 16 18:47:33.673540 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.673517 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk"] Sep 16 18:47:33.676591 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:47:33.676558 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda289a67b_4e5d_49cd_ab8f_0141ee4bb620.slice/crio-3e6a7ed164c61a161c76838429369459c87c2ab6b39bb31ed28b1475b85bf2c4 WatchSource:0}: Error finding container 3e6a7ed164c61a161c76838429369459c87c2ab6b39bb31ed28b1475b85bf2c4: Status 404 returned error can't find the container with id 3e6a7ed164c61a161c76838429369459c87c2ab6b39bb31ed28b1475b85bf2c4 Sep 16 18:47:33.681161 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.681140 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-pprg4"] Sep 16 18:47:33.683802 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:47:33.683780 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26fa1a05_58b1_4135_9703_6e7775f03691.slice/crio-4a382cbda61e83bc6aed103c7ffb0453ecceaa982d4b42fa76d19ed5c663fd79 WatchSource:0}: Error finding container 4a382cbda61e83bc6aed103c7ffb0453ecceaa982d4b42fa76d19ed5c663fd79: Status 404 returned error can't find the container with id 4a382cbda61e83bc6aed103c7ffb0453ecceaa982d4b42fa76d19ed5c663fd79 Sep 16 18:47:33.780882 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.780830 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-dws7b"] Sep 16 18:47:33.974619 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:33.974517 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:33.975039 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.974684 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:47:33.975039 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.974708 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f8c79549-7cwhm: secret "image-registry-tls" not found Sep 16 18:47:33.975039 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:33.974769 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls podName:9777e5e6-2eed-4efd-b9aa-27084ce8debb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.974752672 +0000 UTC m=+142.518058408 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls") pod "image-registry-64f8c79549-7cwhm" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb") : secret "image-registry-tls" not found Sep 16 18:47:34.562405 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:34.562358 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dws7b" event={"ID":"d16ac85c-afc9-484d-8513-fa3f16202d79","Type":"ContainerStarted","Data":"603b9a0862db4ee56aff8478f5fbe810908cd0d1be98e1c3a6c5fa07d863177b"} Sep 16 18:47:34.563915 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:34.563819 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4" event={"ID":"bf8a0586-521c-431c-97f2-6d8faf653e38","Type":"ContainerStarted","Data":"4dd6761dce24662071b370d3ea464a60ea4b8ac0bb04d7fde1d657eafe178d4a"} Sep 16 18:47:34.565950 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:34.565920 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pprg4" event={"ID":"26fa1a05-58b1-4135-9703-6e7775f03691","Type":"ContainerStarted","Data":"29b90a5719235c4c44dc34989ef428c882808fd30548eb6bac4d15d1af2d088e"} Sep 16 18:47:34.566065 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:34.565972 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pprg4" event={"ID":"26fa1a05-58b1-4135-9703-6e7775f03691","Type":"ContainerStarted","Data":"4a382cbda61e83bc6aed103c7ffb0453ecceaa982d4b42fa76d19ed5c663fd79"} Sep 16 18:47:34.568171 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:34.568148 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" event={"ID":"a289a67b-4e5d-49cd-ab8f-0141ee4bb620","Type":"ContainerStarted","Data":"3e6a7ed164c61a161c76838429369459c87c2ab6b39bb31ed28b1475b85bf2c4"} Sep 16 18:47:34.681801 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:34.681135 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:34.681801 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:34.681683 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:34.681801 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:34.681745 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls podName:c017308a-5c89-4157-954c-2bc7f4456b12 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:36.681726166 +0000 UTC m=+144.225031905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-77gx8" (UID: "c017308a-5c89-4157-954c-2bc7f4456b12") : secret "samples-operator-tls" not found Sep 16 18:47:34.984332 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:34.984279 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:34.984855 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:34.984501 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:47:34.984855 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:34.984524 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f8c79549-7cwhm: secret "image-registry-tls" not found Sep 16 18:47:34.984855 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:34.984595 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls podName:9777e5e6-2eed-4efd-b9aa-27084ce8debb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:36.984575539 +0000 UTC m=+144.527881301 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls") pod "image-registry-64f8c79549-7cwhm" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb") : secret "image-registry-tls" not found Sep 16 18:47:35.573597 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:35.573545 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4" event={"ID":"bf8a0586-521c-431c-97f2-6d8faf653e38","Type":"ContainerStarted","Data":"ad8036f62db0b9639db198763f33a3979c08b28e7c1d5e7a886e7098a535c137"} Sep 16 18:47:35.599162 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:35.599108 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-pprg4" podStartSLOduration=2.5990801660000002 podStartE2EDuration="2.599080166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:34.598729598 +0000 UTC m=+142.142035370" watchObservedRunningTime="2026-09-16 18:47:35.599080166 +0000 UTC m=+143.142385926" Sep 16 18:47:35.599720 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:35.599615 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-s8zb4" podStartSLOduration=2.515918857 podStartE2EDuration="3.59960319s" totalImagesPullingTime="1.083684333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:35.598034439 +0000 UTC m=+143.141340198" watchObservedRunningTime="2026-09-16 18:47:35.59960319 +0000 UTC m=+143.142908955" Sep 16 18:47:36.578398 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:36.578359 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" event={"ID":"a289a67b-4e5d-49cd-ab8f-0141ee4bb620","Type":"ContainerStarted","Data":"af25d5966277084dbd9841af6d0e8511d013068c0fa619fea3932d48a00140da"} Sep 16 18:47:36.580235 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:36.580202 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dws7b" event={"ID":"d16ac85c-afc9-484d-8513-fa3f16202d79","Type":"ContainerStarted","Data":"c48ec2c18cb2cd12a867d074385482b615387f00dbc137f1559f413483186707"} Sep 16 18:47:36.626885 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:36.626746 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-dws7b" podStartSLOduration=1.343225582 podStartE2EDuration="3.626723907s" totalImagesPullingTime="2.283498325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:36.625485686 +0000 UTC m=+144.168791446" watchObservedRunningTime="2026-09-16 18:47:36.626723907 +0000 UTC m=+144.170029669" Sep 16 18:47:36.627119 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:36.627079 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" podStartSLOduration=1.234486808 podStartE2EDuration="3.627064282s" totalImagesPullingTime="2.392577474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:36.598144952 +0000 UTC m=+144.141450713" watchObservedRunningTime="2026-09-16 18:47:36.627064282 +0000 UTC m=+144.170370041" Sep 16 18:47:36.696887 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:36.696831 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:36.697072 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:36.696986 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:36.697072 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:36.697065 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls podName:c017308a-5c89-4157-954c-2bc7f4456b12 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:40.697043832 +0000 UTC m=+148.240349589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-77gx8" (UID: "c017308a-5c89-4157-954c-2bc7f4456b12") : secret "samples-operator-tls" not found Sep 16 18:47:37.000337 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.000303 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:37.000522 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:37.000473 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:47:37.000522 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:37.000493 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f8c79549-7cwhm: secret "image-registry-tls" not found Sep 16 18:47:37.000617 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:37.000552 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls podName:9777e5e6-2eed-4efd-b9aa-27084ce8debb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:41.00053443 +0000 UTC m=+148.543840179 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls") pod "image-registry-64f8c79549-7cwhm" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb") : secret "image-registry-tls" not found Sep 16 18:47:37.444445 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.444360 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jkdgk"] Sep 16 18:47:37.450942 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.450912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.455562 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.455534 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 18:47:37.455562 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.455561 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fk2dc\"" Sep 16 18:47:37.455759 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.455543 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 18:47:37.472408 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.472381 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jkdgk"] Sep 16 18:47:37.504276 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.504242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.504456 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.504353 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.504456 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.504385 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdzg\" (UniqueName: \"kubernetes.io/projected/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-api-access-2mdzg\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.504456 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.504412 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/541599ca-7204-4864-a897-ccc1d5d0a5ca-crio-socket\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.504561 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.504510 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/541599ca-7204-4864-a897-ccc1d5d0a5ca-data-volume\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.605116 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.605081 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/541599ca-7204-4864-a897-ccc1d5d0a5ca-data-volume\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.605124 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:37.605229 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:37.605277 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:38.105264336 +0000 UTC m=+145.648570073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.605309 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.605327 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdzg\" (UniqueName: \"kubernetes.io/projected/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-api-access-2mdzg\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.605346 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/541599ca-7204-4864-a897-ccc1d5d0a5ca-crio-socket\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:37.605382 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.605450 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/541599ca-7204-4864-a897-ccc1d5d0a5ca-crio-socket\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:37.605454 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:38.105438763 +0000 UTC m=+145.648744509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : configmap "kube-rbac-proxy" not found Sep 16 18:47:37.605539 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.605459 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/541599ca-7204-4864-a897-ccc1d5d0a5ca-data-volume\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:37.618884 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:37.618840 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdzg\" (UniqueName: \"kubernetes.io/projected/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-api-access-2mdzg\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:38.109369 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:38.109332 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:38.109547 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:38.109431 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:38.109547 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:38.109487 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:38.109547 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:38.109522 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:38.109695 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:38.109552 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:39.109536348 +0000 UTC m=+146.652842085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:38.109695 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:38.109576 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:39.109558702 +0000 UTC m=+146.652864456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : configmap "kube-rbac-proxy" not found Sep 16 18:47:39.117235 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:39.117199 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:39.117749 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:39.117281 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:39.117749 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:39.117356 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:39.117749 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:39.117368 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:39.117749 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:39.117410 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:41.117396176 +0000 UTC m=+148.660701912 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : configmap "kube-rbac-proxy" not found Sep 16 18:47:39.117749 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:39.117422 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:41.117416924 +0000 UTC m=+148.660722661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:39.937545 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:39.937519 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-94bzk_3953984a-4756-4979-9d8d-1f594e70ebd3/dns-node-resolver/0.log" Sep 16 18:47:40.731550 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:40.731516 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:40.731982 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:40.731696 2570 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 16 18:47:40.731982 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:40.731780 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls podName:c017308a-5c89-4157-954c-2bc7f4456b12 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:48.731758551 +0000 UTC m=+156.275064289 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-77gx8" (UID: "c017308a-5c89-4157-954c-2bc7f4456b12") : secret "samples-operator-tls" not found Sep 16 18:47:41.034364 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:41.034260 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:41.034532 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:41.034389 2570 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:47:41.034532 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:41.034400 2570 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64f8c79549-7cwhm: secret "image-registry-tls" not found Sep 16 18:47:41.034532 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:41.034468 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls podName:9777e5e6-2eed-4efd-b9aa-27084ce8debb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:49.034451112 +0000 UTC m=+156.577756850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls") pod "image-registry-64f8c79549-7cwhm" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb") : secret "image-registry-tls" not found Sep 16 18:47:41.135688 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:41.135654 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:41.135920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:41.135717 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:41.135920 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:41.135789 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:41.135920 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:41.135807 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:41.135920 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:41.135892 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.135846739 +0000 UTC m=+152.679152476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:41.136098 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:41.135934 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.135923931 +0000 UTC m=+152.679229671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : configmap "kube-rbac-proxy" not found Sep 16 18:47:41.335450 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:41.335369 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q26jb_bdd72ad2-4fcc-48c2-8bce-1790994113ad/node-ca/0.log" Sep 16 18:47:41.934819 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:41.934784 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-95rsc_22ca6920-60cc-4b51-8e27-f3647ad138e5/serve-healthcheck-canary/0.log" Sep 16 18:47:45.175700 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:45.175663 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:45.176110 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:45.175743 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:45.176110 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:45.175808 2570 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:45.176110 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:45.175817 2570 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:45.176110 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:45.175888 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:53.175851311 +0000 UTC m=+160.719157048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : configmap "kube-rbac-proxy" not found Sep 16 18:47:45.176110 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:45.175911 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls podName:541599ca-7204-4864-a897-ccc1d5d0a5ca nodeName:}" failed. No retries permitted until 2026-09-16 18:47:53.175895761 +0000 UTC m=+160.719201511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jkdgk" (UID: "541599ca-7204-4864-a897-ccc1d5d0a5ca") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:48.809279 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:48.809239 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:48.811760 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:48.811740 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c017308a-5c89-4157-954c-2bc7f4456b12-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-77gx8\" (UID: \"c017308a-5c89-4157-954c-2bc7f4456b12\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:48.875141 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:48.875091 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" Sep 16 18:47:49.114634 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.114538 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:49.117063 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.117034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"image-registry-64f8c79549-7cwhm\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:49.153837 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.153805 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8"] Sep 16 18:47:49.210053 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.210010 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:49.487037 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.487004 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64f8c79549-7cwhm"] Sep 16 18:47:49.490659 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:47:49.490636 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9777e5e6_2eed_4efd_b9aa_27084ce8debb.slice/crio-1b2eed2de9bd839a024207459a098526ef59b541885df38c709ac4bfeec1cc17 WatchSource:0}: Error finding container 1b2eed2de9bd839a024207459a098526ef59b541885df38c709ac4bfeec1cc17: Status 404 returned error can't find the container with id 1b2eed2de9bd839a024207459a098526ef59b541885df38c709ac4bfeec1cc17 Sep 16 18:47:49.626110 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.626072 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" event={"ID":"9777e5e6-2eed-4efd-b9aa-27084ce8debb","Type":"ContainerStarted","Data":"c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db"} Sep 16 18:47:49.626387 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.626128 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" event={"ID":"9777e5e6-2eed-4efd-b9aa-27084ce8debb","Type":"ContainerStarted","Data":"1b2eed2de9bd839a024207459a098526ef59b541885df38c709ac4bfeec1cc17"} Sep 16 18:47:49.626387 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.626192 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:47:49.627389 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.627359 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" event={"ID":"c017308a-5c89-4157-954c-2bc7f4456b12","Type":"ContainerStarted","Data":"2fcbc0d43e82f21dafc64a23f35521e5afa8e43cc9af7bebf7de87c6ed157e56"} Sep 16 18:47:49.656654 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:49.656602 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" podStartSLOduration=16.656583596 podStartE2EDuration="16.656583596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:49.655901355 +0000 UTC m=+157.199207115" watchObservedRunningTime="2026-09-16 18:47:49.656583596 +0000 UTC m=+157.199889389" Sep 16 18:47:51.324080 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:51.324043 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-pvb5t" podUID="fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e" Sep 16 18:47:51.635665 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:51.635572 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvb5t" Sep 16 18:47:51.635665 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:51.635601 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" event={"ID":"c017308a-5c89-4157-954c-2bc7f4456b12","Type":"ContainerStarted","Data":"9eb2e740ed78c319d2b1a8c22c9ca61f9b7c0767e2fb988224c18bbe7e45c23c"} Sep 16 18:47:51.635665 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:51.635659 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" event={"ID":"c017308a-5c89-4157-954c-2bc7f4456b12","Type":"ContainerStarted","Data":"09424473d8ac798177606097c12e03b72966228dc95bade49fd307576b24461c"} Sep 16 18:47:51.689165 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:51.689112 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-77gx8" podStartSLOduration=18.120433281 podStartE2EDuration="19.689096279s" totalImagesPullingTime="1.568662998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:51.687725391 +0000 UTC m=+159.231031150" watchObservedRunningTime="2026-09-16 18:47:51.689096279 +0000 UTC m=+159.232402038" Sep 16 18:47:51.995060 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:47:51.995010 2570 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-mllt4" podUID="f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba" Sep 16 18:47:53.247073 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:53.247037 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:53.247482 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:53.247110 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:53.247581 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:53.247561 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/541599ca-7204-4864-a897-ccc1d5d0a5ca-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:53.249567 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:53.249545 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/541599ca-7204-4864-a897-ccc1d5d0a5ca-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jkdgk\" (UID: \"541599ca-7204-4864-a897-ccc1d5d0a5ca\") " pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:53.359882 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:53.359810 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jkdgk" Sep 16 18:47:53.637882 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:53.637783 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jkdgk"] Sep 16 18:47:53.641830 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:47:53.641804 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541599ca_7204_4864_a897_ccc1d5d0a5ca.slice/crio-c187ecfd1fed9db5be4e48b99de44b2d589487605a2a088b00ea00160c48a8b9 WatchSource:0}: Error finding container c187ecfd1fed9db5be4e48b99de44b2d589487605a2a088b00ea00160c48a8b9: Status 404 returned error can't find the container with id c187ecfd1fed9db5be4e48b99de44b2d589487605a2a088b00ea00160c48a8b9 Sep 16 18:47:54.647339 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:54.647261 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jkdgk" event={"ID":"541599ca-7204-4864-a897-ccc1d5d0a5ca","Type":"ContainerStarted","Data":"8dca3241bc08de0d001b579e6f9dbc2756dcd039cf0f28debe4de68e6e11e2da"} Sep 16 18:47:54.647339 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:54.647323 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jkdgk" event={"ID":"541599ca-7204-4864-a897-ccc1d5d0a5ca","Type":"ContainerStarted","Data":"475e93e23a73240c0e7f1dfaa60411defd55809d3c4f127b437af7a6f7729ca4"} Sep 16 18:47:54.647339 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:54.647342 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jkdgk" event={"ID":"541599ca-7204-4864-a897-ccc1d5d0a5ca","Type":"ContainerStarted","Data":"c187ecfd1fed9db5be4e48b99de44b2d589487605a2a088b00ea00160c48a8b9"} Sep 16 18:47:55.652391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:55.652361 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jkdgk" event={"ID":"541599ca-7204-4864-a897-ccc1d5d0a5ca","Type":"ContainerStarted","Data":"ea9705fe70a98366d2ac34a49b4759a6e7c0b3095eee90653fee7ca48a104d0c"} Sep 16 18:47:55.682406 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:55.682350 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jkdgk" podStartSLOduration=16.871055716 podStartE2EDuration="18.682332845s" totalImagesPullingTime="1.811277129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:55.680595197 +0000 UTC m=+163.223900957" watchObservedRunningTime="2026-09-16 18:47:55.682332845 +0000 UTC m=+163.225638628" Sep 16 18:47:56.273775 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:56.273735 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:47:56.276109 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:56.276081 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e-metrics-tls\") pod \"dns-default-pvb5t\" (UID: \"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e\") " pod="openshift-dns/dns-default-pvb5t" Sep 16 18:47:56.439180 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:56.439152 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgt44\"" Sep 16 18:47:56.447086 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:56.447066 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-pvb5t" Sep 16 18:47:56.732984 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:56.732954 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-pvb5t"] Sep 16 18:47:56.737516 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:47:56.737482 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc62a3fd_a7a7_45dd_bb4d_c2926051eb5e.slice/crio-19b10aac1dc9fb9440b73d7861f4ccdbcfb29821002fc6677c7e3c1c2a486dd3 WatchSource:0}: Error finding container 19b10aac1dc9fb9440b73d7861f4ccdbcfb29821002fc6677c7e3c1c2a486dd3: Status 404 returned error can't find the container with id 19b10aac1dc9fb9440b73d7861f4ccdbcfb29821002fc6677c7e3c1c2a486dd3 Sep 16 18:47:57.660053 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:57.660022 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvb5t" event={"ID":"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e","Type":"ContainerStarted","Data":"19b10aac1dc9fb9440b73d7861f4ccdbcfb29821002fc6677c7e3c1c2a486dd3"} Sep 16 18:47:58.665893 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:58.665833 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvb5t" event={"ID":"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e","Type":"ContainerStarted","Data":"6da07d6bc34935ae6d4b606bd948b207b885131b466a969afaa9f8c520d883a3"} Sep 16 18:47:58.666290 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:58.665927 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-pvb5t" Sep 16 18:47:58.666290 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:58.665955 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-pvb5t" event={"ID":"fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e","Type":"ContainerStarted","Data":"f9451e01cb7c6cbb3cd6fda0d19a0d92fc2de1ef88e411586548bef00e3f0571"} Sep 16 18:47:58.687285 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:58.687231 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-pvb5t" podStartSLOduration=129.280751817 podStartE2EDuration="2m10.687213547s" totalImagesPullingTime="1.40646173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:58.686545353 +0000 UTC m=+166.229851118" watchObservedRunningTime="2026-09-16 18:47:58.687213547 +0000 UTC m=+166.230519305" Sep 16 18:47:59.821119 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:59.821081 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64f8c79549-7cwhm"] Sep 16 18:47:59.864505 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:59.864471 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cb9cc4895-rdq86"] Sep 16 18:47:59.867836 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:59.867818 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:47:59.887740 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:47:59.887714 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cb9cc4895-rdq86"] Sep 16 18:48:00.002706 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.002670 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6867e786-9ad6-4582-92ef-9af500f631f1-registry-certificates\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.002920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.002742 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-registry-tls\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.002920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.002759 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-bound-sa-token\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.002920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.002776 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6867e786-9ad6-4582-92ef-9af500f631f1-image-registry-private-configuration\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.002920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.002798 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzr7q\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-kube-api-access-mzr7q\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.002920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.002812 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6867e786-9ad6-4582-92ef-9af500f631f1-trusted-ca\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.002920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.002875 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6867e786-9ad6-4582-92ef-9af500f631f1-ca-trust-extracted\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.002920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.002916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6867e786-9ad6-4582-92ef-9af500f631f1-installation-pull-secrets\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.103410 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.103331 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6867e786-9ad6-4582-92ef-9af500f631f1-installation-pull-secrets\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.103410 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.103379 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6867e786-9ad6-4582-92ef-9af500f631f1-registry-certificates\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.103623 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.103438 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-registry-tls\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.103623 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.103461 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-bound-sa-token\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.103623 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.103480 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6867e786-9ad6-4582-92ef-9af500f631f1-image-registry-private-configuration\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.103623 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.103504 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzr7q\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-kube-api-access-mzr7q\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.103623 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.103524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6867e786-9ad6-4582-92ef-9af500f631f1-trusted-ca\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.103623 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.103554 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6867e786-9ad6-4582-92ef-9af500f631f1-ca-trust-extracted\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.104218 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.104191 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6867e786-9ad6-4582-92ef-9af500f631f1-ca-trust-extracted\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.104472 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.104453 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6867e786-9ad6-4582-92ef-9af500f631f1-registry-certificates\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.104747 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.104721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6867e786-9ad6-4582-92ef-9af500f631f1-trusted-ca\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.106199 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.106171 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6867e786-9ad6-4582-92ef-9af500f631f1-installation-pull-secrets\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.106592 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.106569 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6867e786-9ad6-4582-92ef-9af500f631f1-image-registry-private-configuration\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.106689 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.106667 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-registry-tls\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.115900 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.115850 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-bound-sa-token\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.117415 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.117393 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzr7q\" (UniqueName: \"kubernetes.io/projected/6867e786-9ad6-4582-92ef-9af500f631f1-kube-api-access-mzr7q\") pod \"image-registry-5cb9cc4895-rdq86\" (UID: \"6867e786-9ad6-4582-92ef-9af500f631f1\") " pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.176413 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.176373 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.466788 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.466749 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cb9cc4895-rdq86"] Sep 16 18:48:00.471336 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:00.471309 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6867e786_9ad6_4582_92ef_9af500f631f1.slice/crio-34537849fee0fd87eccaa63b7ef529c68363484c2902ece1589ca7e39c0a968d WatchSource:0}: Error finding container 34537849fee0fd87eccaa63b7ef529c68363484c2902ece1589ca7e39c0a968d: Status 404 returned error can't find the container with id 34537849fee0fd87eccaa63b7ef529c68363484c2902ece1589ca7e39c0a968d Sep 16 18:48:00.674477 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.674440 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" event={"ID":"6867e786-9ad6-4582-92ef-9af500f631f1","Type":"ContainerStarted","Data":"dda242429bfd667bb515cac6ada4b63a0a9a35c8be63a9b5ea27fcd2a268854d"} Sep 16 18:48:00.674613 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.674499 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" event={"ID":"6867e786-9ad6-4582-92ef-9af500f631f1","Type":"ContainerStarted","Data":"34537849fee0fd87eccaa63b7ef529c68363484c2902ece1589ca7e39c0a968d"} Sep 16 18:48:00.674613 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.674587 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:00.705354 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:00.705298 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" podStartSLOduration=1.705281443 podStartE2EDuration="1.705281443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:00.702050055 +0000 UTC m=+168.245355792" watchObservedRunningTime="2026-09-16 18:48:00.705281443 +0000 UTC m=+168.248587202" Sep 16 18:48:05.965047 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:05.964953 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:48:07.961219 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:07.961188 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt"] Sep 16 18:48:07.965889 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:07.965853 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" Sep 16 18:48:07.968429 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:07.968408 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-pl6zw\"" Sep 16 18:48:07.968586 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:07.968568 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 18:48:07.974704 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:07.974683 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt"] Sep 16 18:48:08.062249 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:08.062215 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba9411be-e65f-428e-86ef-fd4284a981cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lqblt\" (UID: \"ba9411be-e65f-428e-86ef-fd4284a981cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" Sep 16 18:48:08.162564 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:08.162524 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba9411be-e65f-428e-86ef-fd4284a981cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lqblt\" (UID: \"ba9411be-e65f-428e-86ef-fd4284a981cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" Sep 16 18:48:08.165128 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:08.165104 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ba9411be-e65f-428e-86ef-fd4284a981cf-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-lqblt\" (UID: \"ba9411be-e65f-428e-86ef-fd4284a981cf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" Sep 16 18:48:08.275235 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:08.275198 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" Sep 16 18:48:08.545685 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:08.545609 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt"] Sep 16 18:48:08.548489 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:08.548455 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba9411be_e65f_428e_86ef_fd4284a981cf.slice/crio-418574ed047080eb4a3dd719bbe62781ea43b90a3720db058d041c1528f3931f WatchSource:0}: Error finding container 418574ed047080eb4a3dd719bbe62781ea43b90a3720db058d041c1528f3931f: Status 404 returned error can't find the container with id 418574ed047080eb4a3dd719bbe62781ea43b90a3720db058d041c1528f3931f Sep 16 18:48:08.671134 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:08.671102 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-pvb5t" Sep 16 18:48:08.704043 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:08.704015 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" event={"ID":"ba9411be-e65f-428e-86ef-fd4284a981cf","Type":"ContainerStarted","Data":"418574ed047080eb4a3dd719bbe62781ea43b90a3720db058d041c1528f3931f"} Sep 16 18:48:09.709115 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:09.709084 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" event={"ID":"ba9411be-e65f-428e-86ef-fd4284a981cf","Type":"ContainerStarted","Data":"76600833fe69300e906308ba48c80c399fdfd1599f63091844ba6f6c1d3ad899"} Sep 16 18:48:09.709479 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:09.709261 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" Sep 16 18:48:09.714203 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:09.714179 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" Sep 16 18:48:09.728175 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:09.728129 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-lqblt" podStartSLOduration=1.66176718 podStartE2EDuration="2.728112061s" totalImagesPullingTime="1.066344881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:09.726744172 +0000 UTC m=+177.270049931" watchObservedRunningTime="2026-09-16 18:48:09.728112061 +0000 UTC m=+177.271417845" Sep 16 18:48:09.826624 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:09.826591 2570 patch_prober.go:28] interesting pod/image-registry-64f8c79549-7cwhm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 16 18:48:09.826777 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:09.826643 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" podUID="9777e5e6-2eed-4efd-b9aa-27084ce8debb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:48:14.421648 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.421601 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8"] Sep 16 18:48:14.424563 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.424546 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.427248 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.427218 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 18:48:14.427248 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.427229 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 16 18:48:14.427500 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.427486 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 18:48:14.427796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.427782 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hzbhx\"" Sep 16 18:48:14.427960 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.427943 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 16 18:48:14.428599 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.428585 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 18:48:14.438516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.438496 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8"] Sep 16 18:48:14.458076 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.458049 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rwd8j"] Sep 16 18:48:14.460321 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.460306 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.462433 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.462416 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 18:48:14.462647 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.462632 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 18:48:14.462724 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.462688 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 18:48:14.462724 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.462692 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtrnx\"" Sep 16 18:48:14.504256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504228 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.504256 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504261 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee5fee63-a1d4-432a-928b-5946e3253bf4-metrics-client-ca\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504444 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504279 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504444 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504444 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504391 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-textfile\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504444 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504411 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmczb\" (UniqueName: \"kubernetes.io/projected/ee5fee63-a1d4-432a-928b-5946e3253bf4-kube-api-access-mmczb\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504444 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504428 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-tls\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504470 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-sys\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504487 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-wtmp\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504511 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.504604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504563 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7f26\" (UniqueName: \"kubernetes.io/projected/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-kube-api-access-v7f26\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.504604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504577 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-root\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.504604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.504603 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.605758 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605719 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-sys\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.605758 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605753 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-wtmp\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605778 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605800 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7f26\" (UniqueName: \"kubernetes.io/projected/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-kube-api-access-v7f26\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605814 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-root\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605841 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605843 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-sys\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605926 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-root\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605931 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.605930 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-wtmp\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.606019 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee5fee63-a1d4-432a-928b-5946e3253bf4-metrics-client-ca\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606470 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.606065 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606470 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.606097 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606470 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.606160 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-textfile\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606470 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.606185 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmczb\" (UniqueName: \"kubernetes.io/projected/ee5fee63-a1d4-432a-928b-5946e3253bf4-kube-api-access-mmczb\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606470 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.606216 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-tls\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606470 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:48:14.606314 2570 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 18:48:14.606470 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:48:14.606373 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-tls podName:ee5fee63-a1d4-432a-928b-5946e3253bf4 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:15.106353218 +0000 UTC m=+182.649658956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-tls") pod "node-exporter-rwd8j" (UID: "ee5fee63-a1d4-432a-928b-5946e3253bf4") : secret "node-exporter-tls" not found Sep 16 18:48:14.606891 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.606707 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-accelerators-collector-config\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.606891 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.606758 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-textfile\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.607078 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.607033 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.607175 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.607155 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ee5fee63-a1d4-432a-928b-5946e3253bf4-metrics-client-ca\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.608516 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.608480 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.608620 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.608527 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.608806 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.608786 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.620118 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.620089 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmczb\" (UniqueName: \"kubernetes.io/projected/ee5fee63-a1d4-432a-928b-5946e3253bf4-kube-api-access-mmczb\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:14.620306 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.620291 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7f26\" (UniqueName: \"kubernetes.io/projected/798f8fb8-07f0-4c07-84ff-56d9abb88cd9-kube-api-access-v7f26\") pod \"openshift-state-metrics-67dcf8c979-sjmf8\" (UID: \"798f8fb8-07f0-4c07-84ff-56d9abb88cd9\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:14.733995 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:14.733966 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" Sep 16 18:48:15.015135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.015060 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8"] Sep 16 18:48:15.018543 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:15.018521 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798f8fb8_07f0_4c07_84ff_56d9abb88cd9.slice/crio-4b1f00a45151214dc7d34278010de5843260f2cdfa83372db4c89cf7377e4dbb WatchSource:0}: Error finding container 4b1f00a45151214dc7d34278010de5843260f2cdfa83372db4c89cf7377e4dbb: Status 404 returned error can't find the container with id 4b1f00a45151214dc7d34278010de5843260f2cdfa83372db4c89cf7377e4dbb Sep 16 18:48:15.109141 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.109116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-tls\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:15.111480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.111442 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/ee5fee63-a1d4-432a-928b-5946e3253bf4-node-exporter-tls\") pod \"node-exporter-rwd8j\" (UID: \"ee5fee63-a1d4-432a-928b-5946e3253bf4\") " pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:15.368784 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.368696 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rwd8j" Sep 16 18:48:15.377991 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:15.377963 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5fee63_a1d4_432a_928b_5946e3253bf4.slice/crio-a70f2491f7fb8902500a251d714cce9c20d416777166a9d8c9cdfed0d64c26f5 WatchSource:0}: Error finding container a70f2491f7fb8902500a251d714cce9c20d416777166a9d8c9cdfed0d64c26f5: Status 404 returned error can't find the container with id a70f2491f7fb8902500a251d714cce9c20d416777166a9d8c9cdfed0d64c26f5 Sep 16 18:48:15.731763 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.731725 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" event={"ID":"798f8fb8-07f0-4c07-84ff-56d9abb88cd9","Type":"ContainerStarted","Data":"de8c4463d9ff9eccaa7b62cac08c292058a4de55ef25ae05b6414b225f5028d7"} Sep 16 18:48:15.732214 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.731793 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" event={"ID":"798f8fb8-07f0-4c07-84ff-56d9abb88cd9","Type":"ContainerStarted","Data":"a37013c49df7cc1887c80381e92935c951d670e2334a01d0258f144be915ecc9"} Sep 16 18:48:15.732214 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.731820 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" event={"ID":"798f8fb8-07f0-4c07-84ff-56d9abb88cd9","Type":"ContainerStarted","Data":"4b1f00a45151214dc7d34278010de5843260f2cdfa83372db4c89cf7377e4dbb"} Sep 16 18:48:15.732990 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.732967 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwd8j" event={"ID":"ee5fee63-a1d4-432a-928b-5946e3253bf4","Type":"ContainerStarted","Data":"a70f2491f7fb8902500a251d714cce9c20d416777166a9d8c9cdfed0d64c26f5"} Sep 16 18:48:15.918250 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.918203 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:48:15.921400 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.921377 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:15.924207 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.924181 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 18:48:15.924319 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.924211 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kd2hz\"" Sep 16 18:48:15.924319 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.924184 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 18:48:15.924637 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.924620 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 18:48:15.924711 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.924691 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 18:48:15.924774 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.924620 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 18:48:15.924927 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.924910 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 18:48:15.924983 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.924916 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 18:48:15.925017 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.925005 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 18:48:15.925330 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.925310 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 18:48:15.943579 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:15.943548 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:48:16.015819 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.015641 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-out\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.015819 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.015697 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.015819 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.015738 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.015819 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.015796 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.015819 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.015819 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n29jt\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-kube-api-access-n29jt\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.016164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.015924 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.016164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.015985 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.016164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.016030 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.016164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.016063 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-web-config\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.016164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.016090 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.016164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.016129 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.016164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.016156 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.016534 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.016203 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117631 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117593 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117631 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117631 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n29jt\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-kube-api-access-n29jt\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117658 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117692 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117731 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117761 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-web-config\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117785 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117811 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.117899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117828 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.118322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117904 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.118322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117933 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-out\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.118322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117959 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.118322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.117997 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.118322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.118260 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.121527 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.121217 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-volume\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.121527 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.121416 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.121527 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.121434 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.121792 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.121620 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-web-config\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.121792 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.121708 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.122238 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.122210 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.122437 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.122384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.122763 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.122746 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.122900 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.122848 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.123660 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.123641 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-out\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.123799 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.123773 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.132672 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.132640 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n29jt\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-kube-api-access-n29jt\") pod \"alertmanager-main-0\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.234107 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.234070 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:48:16.739281 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.739242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" event={"ID":"798f8fb8-07f0-4c07-84ff-56d9abb88cd9","Type":"ContainerStarted","Data":"f93ebbe1c1267d73e7de7c7e99ceb78ab5594f3b5f13f96df766abb026329d51"} Sep 16 18:48:16.740900 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.740856 2570 generic.go:356] "Generic (PLEG): container finished" podID="ee5fee63-a1d4-432a-928b-5946e3253bf4" containerID="8663f4e723d4c4a54f237b417af983a0fc4175c38958adf85d0858bf574ca24d" exitCode=0 Sep 16 18:48:16.741023 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.740943 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwd8j" event={"ID":"ee5fee63-a1d4-432a-928b-5946e3253bf4","Type":"ContainerDied","Data":"8663f4e723d4c4a54f237b417af983a0fc4175c38958adf85d0858bf574ca24d"} Sep 16 18:48:16.769215 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.769166 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-sjmf8" podStartSLOduration=1.383558063 podStartE2EDuration="2.769148715s" totalImagesPullingTime="1.385590652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:16.768716299 +0000 UTC m=+184.312022058" watchObservedRunningTime="2026-09-16 18:48:16.769148715 +0000 UTC m=+184.312454509" Sep 16 18:48:16.807105 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:16.807075 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:48:16.808348 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:16.808219 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3b8782_c5b8_4cb5_b635_043b63fad944.slice/crio-c86f4904e84ef63a6cd2980317c15814369e14f2bd12a314b5895a54449fa791 WatchSource:0}: Error finding container c86f4904e84ef63a6cd2980317c15814369e14f2bd12a314b5895a54449fa791: Status 404 returned error can't find the container with id c86f4904e84ef63a6cd2980317c15814369e14f2bd12a314b5895a54449fa791 Sep 16 18:48:17.747907 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:17.747847 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwd8j" event={"ID":"ee5fee63-a1d4-432a-928b-5946e3253bf4","Type":"ContainerStarted","Data":"dd084c9f4349bf14f686aa6f717729aa815de38f985185050c5918870865c4d0"} Sep 16 18:48:17.748358 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:17.747940 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rwd8j" event={"ID":"ee5fee63-a1d4-432a-928b-5946e3253bf4","Type":"ContainerStarted","Data":"a82f7d79d3c0263fa850433cb08531dd25e22955d2a81ae3cd871c386a7ae25c"} Sep 16 18:48:17.749397 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:17.749371 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerStarted","Data":"c86f4904e84ef63a6cd2980317c15814369e14f2bd12a314b5895a54449fa791"} Sep 16 18:48:17.780720 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:17.780655 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rwd8j" podStartSLOduration=2.574248119 podStartE2EDuration="3.780631657s" totalImagesPullingTime="1.133227276s" totalInitContainerRuntime="73.156262ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:17.778693109 +0000 UTC m=+185.321998893" watchObservedRunningTime="2026-09-16 18:48:17.780631657 +0000 UTC m=+185.323937417" Sep 16 18:48:18.754727 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.754694 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerID="ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c" exitCode=0 Sep 16 18:48:18.755126 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.754786 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerDied","Data":"ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c"} Sep 16 18:48:18.848788 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.848730 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69dd66586b-btdkd"] Sep 16 18:48:18.851605 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.851589 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:18.854245 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.854225 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 18:48:18.854520 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.854493 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 16 18:48:18.854520 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.854511 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 16 18:48:18.854670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.854660 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 16 18:48:18.854939 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.854922 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-cd5h7\"" Sep 16 18:48:18.857456 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.857439 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-7g4r94evv66tg\"" Sep 16 18:48:18.864060 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.864036 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69dd66586b-btdkd"] Sep 16 18:48:18.943745 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.943701 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-secret-metrics-server-client-certs\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:18.943967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.943767 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5de4aef8-79a8-4987-8358-1cc178e38454-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:18.943967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.943850 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5de4aef8-79a8-4987-8358-1cc178e38454-metrics-server-audit-profiles\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:18.943967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.943894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-secret-metrics-server-tls\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:18.943967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.943921 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkp4r\" (UniqueName: \"kubernetes.io/projected/5de4aef8-79a8-4987-8358-1cc178e38454-kube-api-access-nkp4r\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:18.944135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.943968 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-client-ca-bundle\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:18.944135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:18.943998 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5de4aef8-79a8-4987-8358-1cc178e38454-audit-log\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.045375 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.045291 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5de4aef8-79a8-4987-8358-1cc178e38454-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.045375 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.045335 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5de4aef8-79a8-4987-8358-1cc178e38454-metrics-server-audit-profiles\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.045375 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.045354 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-secret-metrics-server-tls\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.045638 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.045382 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkp4r\" (UniqueName: \"kubernetes.io/projected/5de4aef8-79a8-4987-8358-1cc178e38454-kube-api-access-nkp4r\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.045638 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.045424 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-client-ca-bundle\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.045638 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.045458 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5de4aef8-79a8-4987-8358-1cc178e38454-audit-log\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.045638 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.045509 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-secret-metrics-server-client-certs\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.045952 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.045928 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/5de4aef8-79a8-4987-8358-1cc178e38454-audit-log\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.046200 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.046181 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5de4aef8-79a8-4987-8358-1cc178e38454-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.046466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.046398 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/5de4aef8-79a8-4987-8358-1cc178e38454-metrics-server-audit-profiles\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.048199 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.048172 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-secret-metrics-server-client-certs\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.048533 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.048511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-client-ca-bundle\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.048608 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.048543 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/5de4aef8-79a8-4987-8358-1cc178e38454-secret-metrics-server-tls\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.055017 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.054990 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkp4r\" (UniqueName: \"kubernetes.io/projected/5de4aef8-79a8-4987-8358-1cc178e38454-kube-api-access-nkp4r\") pod \"metrics-server-69dd66586b-btdkd\" (UID: \"5de4aef8-79a8-4987-8358-1cc178e38454\") " pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.155566 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.155524 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26"] Sep 16 18:48:19.159182 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.159160 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:19.160305 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.160285 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:19.162176 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.162153 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-smchq\"" Sep 16 18:48:19.162710 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.162691 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 16 18:48:19.168819 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.168793 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26"] Sep 16 18:48:19.247346 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.247311 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19beda34-68cc-4379-8bf4-7d41f797995e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4v26\" (UID: \"19beda34-68cc-4379-8bf4-7d41f797995e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:19.247498 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.247360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf759\" (UniqueName: \"kubernetes.io/projected/19beda34-68cc-4379-8bf4-7d41f797995e-kube-api-access-hf759\") pod \"monitoring-plugin-5d85778b9d-j4v26\" (UID: \"19beda34-68cc-4379-8bf4-7d41f797995e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:19.348695 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.348606 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19beda34-68cc-4379-8bf4-7d41f797995e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4v26\" (UID: \"19beda34-68cc-4379-8bf4-7d41f797995e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:19.348695 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.348655 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf759\" (UniqueName: \"kubernetes.io/projected/19beda34-68cc-4379-8bf4-7d41f797995e-kube-api-access-hf759\") pod \"monitoring-plugin-5d85778b9d-j4v26\" (UID: \"19beda34-68cc-4379-8bf4-7d41f797995e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:19.348977 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:48:19.348783 2570 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 16 18:48:19.348977 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:48:19.348880 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19beda34-68cc-4379-8bf4-7d41f797995e-monitoring-plugin-cert podName:19beda34-68cc-4379-8bf4-7d41f797995e nodeName:}" failed. No retries permitted until 2026-09-16 18:48:19.848840017 +0000 UTC m=+187.392145773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/19beda34-68cc-4379-8bf4-7d41f797995e-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-j4v26" (UID: "19beda34-68cc-4379-8bf4-7d41f797995e") : secret "monitoring-plugin-cert" not found Sep 16 18:48:19.364541 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.364511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf759\" (UniqueName: \"kubernetes.io/projected/19beda34-68cc-4379-8bf4-7d41f797995e-kube-api-access-hf759\") pod \"monitoring-plugin-5d85778b9d-j4v26\" (UID: \"19beda34-68cc-4379-8bf4-7d41f797995e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:19.447229 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.447191 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69dd66586b-btdkd"] Sep 16 18:48:19.450302 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:19.450278 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de4aef8_79a8_4987_8358_1cc178e38454.slice/crio-41ad44fc6abd39464bdb2d65df3ab4d016787bfe4f99273427ae7bd131384fb7 WatchSource:0}: Error finding container 41ad44fc6abd39464bdb2d65df3ab4d016787bfe4f99273427ae7bd131384fb7: Status 404 returned error can't find the container with id 41ad44fc6abd39464bdb2d65df3ab4d016787bfe4f99273427ae7bd131384fb7 Sep 16 18:48:19.760649 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.760600 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" event={"ID":"5de4aef8-79a8-4987-8358-1cc178e38454","Type":"ContainerStarted","Data":"41ad44fc6abd39464bdb2d65df3ab4d016787bfe4f99273427ae7bd131384fb7"} Sep 16 18:48:19.826150 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.826119 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:48:19.852535 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.852491 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19beda34-68cc-4379-8bf4-7d41f797995e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4v26\" (UID: \"19beda34-68cc-4379-8bf4-7d41f797995e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:19.855319 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:19.855290 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/19beda34-68cc-4379-8bf4-7d41f797995e-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-j4v26\" (UID: \"19beda34-68cc-4379-8bf4-7d41f797995e\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:20.070000 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:20.069926 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:20.409408 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:20.409372 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26"] Sep 16 18:48:20.414019 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:20.413990 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19beda34_68cc_4379_8bf4_7d41f797995e.slice/crio-da903ef7b2436266dc5461f52e20325cc357a95fdd78b71e200b63a352c9d73d WatchSource:0}: Error finding container da903ef7b2436266dc5461f52e20325cc357a95fdd78b71e200b63a352c9d73d: Status 404 returned error can't find the container with id da903ef7b2436266dc5461f52e20325cc357a95fdd78b71e200b63a352c9d73d Sep 16 18:48:20.765906 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:20.765845 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" event={"ID":"19beda34-68cc-4379-8bf4-7d41f797995e","Type":"ContainerStarted","Data":"da903ef7b2436266dc5461f52e20325cc357a95fdd78b71e200b63a352c9d73d"} Sep 16 18:48:20.769442 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:20.769406 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerStarted","Data":"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab"} Sep 16 18:48:20.769569 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:20.769466 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerStarted","Data":"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117"} Sep 16 18:48:20.769569 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:20.769494 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerStarted","Data":"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4"} Sep 16 18:48:20.769569 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:20.769531 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerStarted","Data":"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133"} Sep 16 18:48:20.769569 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:20.769565 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerStarted","Data":"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57"} Sep 16 18:48:21.681511 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:21.681428 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cb9cc4895-rdq86" Sep 16 18:48:21.777101 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:21.777059 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" event={"ID":"5de4aef8-79a8-4987-8358-1cc178e38454","Type":"ContainerStarted","Data":"c18f5d641a241aee8ba3bf3a732664952ce6ea4f075fc96901128496d538bb51"} Sep 16 18:48:21.800902 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:21.800821 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" podStartSLOduration=1.992654178 podStartE2EDuration="3.800801452s" totalImagesPullingTime="1.808147274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:21.799525202 +0000 UTC m=+189.342830961" watchObservedRunningTime="2026-09-16 18:48:21.800801452 +0000 UTC m=+189.344107211" Sep 16 18:48:22.784844 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:22.784801 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerStarted","Data":"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857"} Sep 16 18:48:22.786351 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:22.786328 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" event={"ID":"19beda34-68cc-4379-8bf4-7d41f797995e","Type":"ContainerStarted","Data":"2ff2607955bda988e83d984805a1186954a888c1a95cb8d5f1baba6852c8f36a"} Sep 16 18:48:22.786510 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:22.786491 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:22.791285 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:22.791260 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" Sep 16 18:48:22.834562 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:22.834513 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.875252359 podStartE2EDuration="7.834496163s" totalImagesPullingTime="3.943477411s" totalInitContainerRuntime="15.766393ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:22.834039492 +0000 UTC m=+190.377345250" watchObservedRunningTime="2026-09-16 18:48:22.834496163 +0000 UTC m=+190.377801921" Sep 16 18:48:24.840778 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:24.840735 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" podUID="9777e5e6-2eed-4efd-b9aa-27084ce8debb" containerName="registry" containerID="cri-o://c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db" gracePeriod=30 Sep 16 18:48:25.085105 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.085079 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:48:25.118094 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.117986 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-j4v26" podStartSLOduration=4.659018768 podStartE2EDuration="6.117968358s" totalImagesPullingTime="1.45894959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:22.858371393 +0000 UTC m=+190.401677149" watchObservedRunningTime="2026-09-16 18:48:25.117968358 +0000 UTC m=+192.661274117" Sep 16 18:48:25.208029 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.207990 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-image-registry-private-configuration\") pod \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " Sep 16 18:48:25.208229 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208047 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-certificates\") pod \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " Sep 16 18:48:25.208229 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208084 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-installation-pull-secrets\") pod \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " Sep 16 18:48:25.208229 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208130 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-trusted-ca\") pod \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " Sep 16 18:48:25.208452 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208256 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9777e5e6-2eed-4efd-b9aa-27084ce8debb-ca-trust-extracted\") pod \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " Sep 16 18:48:25.208452 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208304 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") pod \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " Sep 16 18:48:25.208452 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208356 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-bound-sa-token\") pod \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " Sep 16 18:48:25.208452 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208382 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkphs\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-kube-api-access-bkphs\") pod \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\" (UID: \"9777e5e6-2eed-4efd-b9aa-27084ce8debb\") " Sep 16 18:48:25.208901 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208629 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9777e5e6-2eed-4efd-b9aa-27084ce8debb" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:48:25.208901 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.208828 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9777e5e6-2eed-4efd-b9aa-27084ce8debb" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:48:25.211371 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.211324 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-kube-api-access-bkphs" (OuterVolumeSpecName: "kube-api-access-bkphs") pod "9777e5e6-2eed-4efd-b9aa-27084ce8debb" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb"). InnerVolumeSpecName "kube-api-access-bkphs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:25.211371 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.211340 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9777e5e6-2eed-4efd-b9aa-27084ce8debb" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:48:25.211532 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.211486 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9777e5e6-2eed-4efd-b9aa-27084ce8debb" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:48:25.211572 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.211553 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9777e5e6-2eed-4efd-b9aa-27084ce8debb" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:25.211607 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.211560 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9777e5e6-2eed-4efd-b9aa-27084ce8debb" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:25.218010 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.217987 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9777e5e6-2eed-4efd-b9aa-27084ce8debb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9777e5e6-2eed-4efd-b9aa-27084ce8debb" (UID: "9777e5e6-2eed-4efd-b9aa-27084ce8debb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:48:25.309959 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.309925 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-trusted-ca\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:48:25.309959 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.309952 2570 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9777e5e6-2eed-4efd-b9aa-27084ce8debb-ca-trust-extracted\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:48:25.309959 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.309962 2570 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:48:25.309959 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.309971 2570 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-bound-sa-token\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:48:25.310212 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.309979 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkphs\" (UniqueName: \"kubernetes.io/projected/9777e5e6-2eed-4efd-b9aa-27084ce8debb-kube-api-access-bkphs\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:48:25.310212 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.309987 2570 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-image-registry-private-configuration\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:48:25.310212 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.309996 2570 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9777e5e6-2eed-4efd-b9aa-27084ce8debb-registry-certificates\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:48:25.310212 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.310004 2570 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9777e5e6-2eed-4efd-b9aa-27084ce8debb-installation-pull-secrets\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:48:25.800063 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.800033 2570 generic.go:356] "Generic (PLEG): container finished" podID="9777e5e6-2eed-4efd-b9aa-27084ce8debb" containerID="c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db" exitCode=0 Sep 16 18:48:25.800237 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.800080 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" Sep 16 18:48:25.800237 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.800129 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" event={"ID":"9777e5e6-2eed-4efd-b9aa-27084ce8debb","Type":"ContainerDied","Data":"c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db"} Sep 16 18:48:25.800237 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.800181 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64f8c79549-7cwhm" event={"ID":"9777e5e6-2eed-4efd-b9aa-27084ce8debb","Type":"ContainerDied","Data":"1b2eed2de9bd839a024207459a098526ef59b541885df38c709ac4bfeec1cc17"} Sep 16 18:48:25.800237 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.800209 2570 scope.go:117] "RemoveContainer" containerID="c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db" Sep 16 18:48:25.808890 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.808849 2570 scope.go:117] "RemoveContainer" containerID="c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db" Sep 16 18:48:25.809163 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:48:25.809143 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db\": container with ID starting with c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db not found: ID does not exist" containerID="c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db" Sep 16 18:48:25.809211 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.809172 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db"} err="failed to get container status \"c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db\": rpc error: code = NotFound desc = could not find container \"c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db\": container with ID starting with c6cb859a960f08ba1c32d31655263c13e322bcfdee936795702038f7e67c61db not found: ID does not exist" Sep 16 18:48:25.822179 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.822149 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64f8c79549-7cwhm"] Sep 16 18:48:25.830645 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:25.830623 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64f8c79549-7cwhm"] Sep 16 18:48:26.970611 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:26.970582 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9777e5e6-2eed-4efd-b9aa-27084ce8debb" path="/var/lib/kubelet/pods/9777e5e6-2eed-4efd-b9aa-27084ce8debb/volumes" Sep 16 18:48:28.182032 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.182002 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-qxsx2"] Sep 16 18:48:28.182413 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.182311 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9777e5e6-2eed-4efd-b9aa-27084ce8debb" containerName="registry" Sep 16 18:48:28.182413 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.182322 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="9777e5e6-2eed-4efd-b9aa-27084ce8debb" containerName="registry" Sep 16 18:48:28.182413 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.182372 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="9777e5e6-2eed-4efd-b9aa-27084ce8debb" containerName="registry" Sep 16 18:48:28.186646 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.186624 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qxsx2" Sep 16 18:48:28.189398 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.189376 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 18:48:28.189874 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.189849 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-js4wm\"" Sep 16 18:48:28.196421 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.196400 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 18:48:28.214490 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.214456 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qxsx2"] Sep 16 18:48:28.233358 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.233322 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snm8w\" (UniqueName: \"kubernetes.io/projected/feed169e-b4a9-476f-82a3-250cc36f122e-kube-api-access-snm8w\") pod \"downloads-76f99584f4-qxsx2\" (UID: \"feed169e-b4a9-476f-82a3-250cc36f122e\") " pod="openshift-console/downloads-76f99584f4-qxsx2" Sep 16 18:48:28.334005 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.333962 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snm8w\" (UniqueName: \"kubernetes.io/projected/feed169e-b4a9-476f-82a3-250cc36f122e-kube-api-access-snm8w\") pod \"downloads-76f99584f4-qxsx2\" (UID: \"feed169e-b4a9-476f-82a3-250cc36f122e\") " pod="openshift-console/downloads-76f99584f4-qxsx2" Sep 16 18:48:28.352353 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.352317 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snm8w\" (UniqueName: \"kubernetes.io/projected/feed169e-b4a9-476f-82a3-250cc36f122e-kube-api-access-snm8w\") pod \"downloads-76f99584f4-qxsx2\" (UID: \"feed169e-b4a9-476f-82a3-250cc36f122e\") " pod="openshift-console/downloads-76f99584f4-qxsx2" Sep 16 18:48:28.496142 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.496093 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-qxsx2" Sep 16 18:48:28.774743 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.774665 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-qxsx2"] Sep 16 18:48:28.778833 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:28.778797 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeed169e_b4a9_476f_82a3_250cc36f122e.slice/crio-22b676bd565dc93814091b30db27f6e7923bc0ee63e8f64532aceab74a48a71d WatchSource:0}: Error finding container 22b676bd565dc93814091b30db27f6e7923bc0ee63e8f64532aceab74a48a71d: Status 404 returned error can't find the container with id 22b676bd565dc93814091b30db27f6e7923bc0ee63e8f64532aceab74a48a71d Sep 16 18:48:28.812944 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:28.812908 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qxsx2" event={"ID":"feed169e-b4a9-476f-82a3-250cc36f122e","Type":"ContainerStarted","Data":"22b676bd565dc93814091b30db27f6e7923bc0ee63e8f64532aceab74a48a71d"} Sep 16 18:48:37.832438 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.832403 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-668d8b7689-bqk9w"] Sep 16 18:48:37.835193 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.835168 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:37.837677 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.837651 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 16 18:48:37.837804 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.837677 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-88nh4\"" Sep 16 18:48:37.837804 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.837762 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 16 18:48:37.838300 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.838275 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 16 18:48:37.838612 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.838594 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 16 18:48:37.838716 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.838626 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 16 18:48:37.847367 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.847345 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668d8b7689-bqk9w"] Sep 16 18:48:37.928467 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.928434 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-serving-cert\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:37.928671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.928484 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-config\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:37.928671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.928525 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-oauth-serving-cert\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:37.928671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.928552 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8lbl\" (UniqueName: \"kubernetes.io/projected/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-kube-api-access-v8lbl\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:37.928671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.928635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-service-ca\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:37.928876 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:37.928675 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-oauth-config\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.029684 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.029651 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-serving-cert\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.029684 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.029698 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-config\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.029968 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.029736 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-oauth-serving-cert\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.029968 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.029765 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lbl\" (UniqueName: \"kubernetes.io/projected/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-kube-api-access-v8lbl\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.029968 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.029807 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-service-ca\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.029968 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.029836 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-oauth-config\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.030767 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.030536 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-config\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.030767 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.030613 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-service-ca\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.030767 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.030626 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-oauth-serving-cert\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.032563 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.032540 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-oauth-config\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.032692 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.032671 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-serving-cert\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.040009 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.039984 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lbl\" (UniqueName: \"kubernetes.io/projected/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-kube-api-access-v8lbl\") pod \"console-668d8b7689-bqk9w\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:38.147299 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:38.147212 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:39.160579 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:39.160542 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:39.160998 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:39.160615 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:40.042841 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.042805 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-947d58874-rtnpp"] Sep 16 18:48:40.045480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.045456 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.054438 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.054408 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 16 18:48:40.059830 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.059804 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-947d58874-rtnpp"] Sep 16 18:48:40.152204 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.152163 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-service-ca\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.152204 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.152208 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmj4v\" (UniqueName: \"kubernetes.io/projected/823760f5-fcef-47c4-a1cf-a35209fba9fc-kube-api-access-kmj4v\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.152449 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.152242 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-serving-cert\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.152449 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.152360 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-trusted-ca-bundle\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.152449 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.152429 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-oauth-config\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.152573 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.152461 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-config\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.152573 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.152531 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-oauth-serving-cert\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.253584 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.253553 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-oauth-serving-cert\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254104 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.253599 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-service-ca\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254104 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.253624 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmj4v\" (UniqueName: \"kubernetes.io/projected/823760f5-fcef-47c4-a1cf-a35209fba9fc-kube-api-access-kmj4v\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254104 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.253652 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-serving-cert\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254104 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.253685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-trusted-ca-bundle\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254104 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.253715 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-oauth-config\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254104 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.253734 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-config\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254420 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.254334 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-config\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254474 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.254430 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-oauth-serving-cert\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.254825 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.254780 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-service-ca\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.255409 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.255384 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-trusted-ca-bundle\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.257515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.257490 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-oauth-config\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.257636 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.257619 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-serving-cert\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.264999 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.264920 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmj4v\" (UniqueName: \"kubernetes.io/projected/823760f5-fcef-47c4-a1cf-a35209fba9fc-kube-api-access-kmj4v\") pod \"console-947d58874-rtnpp\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:40.363101 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:40.363010 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:44.819303 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:44.819144 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-947d58874-rtnpp"] Sep 16 18:48:44.822438 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:44.822137 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-668d8b7689-bqk9w"] Sep 16 18:48:44.823153 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:44.823124 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823760f5_fcef_47c4_a1cf_a35209fba9fc.slice/crio-f3e7cfcc6c40d66597d34e5e62ccc3aa9ba8ba7db2d7a681bdcaeef93de68e3e WatchSource:0}: Error finding container f3e7cfcc6c40d66597d34e5e62ccc3aa9ba8ba7db2d7a681bdcaeef93de68e3e: Status 404 returned error can't find the container with id f3e7cfcc6c40d66597d34e5e62ccc3aa9ba8ba7db2d7a681bdcaeef93de68e3e Sep 16 18:48:44.825110 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:48:44.825065 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee1dc8b_8d6c_45d7_94d2_744f9b4016be.slice/crio-abc57e6ac967a9be522762daec18dca1f478c1fd97852657f48bf7ac8f97fee9 WatchSource:0}: Error finding container abc57e6ac967a9be522762daec18dca1f478c1fd97852657f48bf7ac8f97fee9: Status 404 returned error can't find the container with id abc57e6ac967a9be522762daec18dca1f478c1fd97852657f48bf7ac8f97fee9 Sep 16 18:48:44.883614 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:44.883582 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668d8b7689-bqk9w" event={"ID":"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be","Type":"ContainerStarted","Data":"abc57e6ac967a9be522762daec18dca1f478c1fd97852657f48bf7ac8f97fee9"} Sep 16 18:48:44.884996 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:44.884961 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-947d58874-rtnpp" event={"ID":"823760f5-fcef-47c4-a1cf-a35209fba9fc","Type":"ContainerStarted","Data":"f3e7cfcc6c40d66597d34e5e62ccc3aa9ba8ba7db2d7a681bdcaeef93de68e3e"} Sep 16 18:48:44.886807 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:44.886778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-qxsx2" event={"ID":"feed169e-b4a9-476f-82a3-250cc36f122e","Type":"ContainerStarted","Data":"31c9eaecd7b58c10f6754c70a2e1a3c22c090f91ee143a5a2e230748bf94a3d4"} Sep 16 18:48:44.887099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:44.887078 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-qxsx2" Sep 16 18:48:44.895178 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:44.895158 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-qxsx2" Sep 16 18:48:44.906837 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:44.906775 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-qxsx2" podStartSLOduration=1.042076853 podStartE2EDuration="16.906759144s" totalImagesPullingTime="15.864682291s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:44.905332747 +0000 UTC m=+212.448638509" watchObservedRunningTime="2026-09-16 18:48:44.906759144 +0000 UTC m=+212.450064902" Sep 16 18:48:46.901927 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:46.901390 2570 generic.go:356] "Generic (PLEG): container finished" podID="a289a67b-4e5d-49cd-ab8f-0141ee4bb620" containerID="af25d5966277084dbd9841af6d0e8511d013068c0fa619fea3932d48a00140da" exitCode=0 Sep 16 18:48:46.901927 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:46.901503 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" event={"ID":"a289a67b-4e5d-49cd-ab8f-0141ee4bb620","Type":"ContainerDied","Data":"af25d5966277084dbd9841af6d0e8511d013068c0fa619fea3932d48a00140da"} Sep 16 18:48:46.903092 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:46.902571 2570 scope.go:117] "RemoveContainer" containerID="af25d5966277084dbd9841af6d0e8511d013068c0fa619fea3932d48a00140da" Sep 16 18:48:46.907561 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:46.906673 2570 generic.go:356] "Generic (PLEG): container finished" podID="d16ac85c-afc9-484d-8513-fa3f16202d79" containerID="c48ec2c18cb2cd12a867d074385482b615387f00dbc137f1559f413483186707" exitCode=0 Sep 16 18:48:46.907561 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:46.907058 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dws7b" event={"ID":"d16ac85c-afc9-484d-8513-fa3f16202d79","Type":"ContainerDied","Data":"c48ec2c18cb2cd12a867d074385482b615387f00dbc137f1559f413483186707"} Sep 16 18:48:46.907561 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:46.907379 2570 scope.go:117] "RemoveContainer" containerID="c48ec2c18cb2cd12a867d074385482b615387f00dbc137f1559f413483186707" Sep 16 18:48:47.919087 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:47.919049 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-dws7b" event={"ID":"d16ac85c-afc9-484d-8513-fa3f16202d79","Type":"ContainerStarted","Data":"9c85ed1d8fa860732e81ba921dee0ebf1c241f0935370639a8596ebdaeced1c2"} Sep 16 18:48:47.921487 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:47.921422 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-2jzhk" event={"ID":"a289a67b-4e5d-49cd-ab8f-0141ee4bb620","Type":"ContainerStarted","Data":"c34af527e3a8eb7984f40c40c3c1a7a6ab4f1c0efcfc25ee9a8e9bb9fe7c8b23"} Sep 16 18:48:48.068729 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:48.068385 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3b8782-c5b8-4cb5-b635-043b63fad944/init-config-reloader/0.log" Sep 16 18:48:48.077391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:48.077253 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3b8782-c5b8-4cb5-b635-043b63fad944/alertmanager/0.log" Sep 16 18:48:48.088562 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:48.088512 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3b8782-c5b8-4cb5-b635-043b63fad944/config-reloader/0.log" Sep 16 18:48:48.238500 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:48.238425 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3b8782-c5b8-4cb5-b635-043b63fad944/kube-rbac-proxy-web/0.log" Sep 16 18:48:48.440373 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:48.440345 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3b8782-c5b8-4cb5-b635-043b63fad944/kube-rbac-proxy/0.log" Sep 16 18:48:48.637260 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:48.637151 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3b8782-c5b8-4cb5-b635-043b63fad944/kube-rbac-proxy-metric/0.log" Sep 16 18:48:48.835680 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:48.835638 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3f3b8782-c5b8-4cb5-b635-043b63fad944/prom-label-proxy/0.log" Sep 16 18:48:49.836636 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:49.836554 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69dd66586b-btdkd_5de4aef8-79a8-4987-8358-1cc178e38454/metrics-server/0.log" Sep 16 18:48:49.933707 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:49.933667 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668d8b7689-bqk9w" event={"ID":"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be","Type":"ContainerStarted","Data":"5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d"} Sep 16 18:48:49.935838 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:49.935807 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-947d58874-rtnpp" event={"ID":"823760f5-fcef-47c4-a1cf-a35209fba9fc","Type":"ContainerStarted","Data":"519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12"} Sep 16 18:48:49.954918 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:49.954852 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-668d8b7689-bqk9w" podStartSLOduration=8.319553656 podStartE2EDuration="12.954832333s" totalImagesPullingTime="4.635278677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:49.952637467 +0000 UTC m=+217.495943260" watchObservedRunningTime="2026-09-16 18:48:49.954832333 +0000 UTC m=+217.498138099" Sep 16 18:48:49.970992 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:49.970945 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-947d58874-rtnpp" podStartSLOduration=5.485832642 podStartE2EDuration="9.970930389s" totalImagesPullingTime="4.485097747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:49.96928433 +0000 UTC m=+217.512590091" watchObservedRunningTime="2026-09-16 18:48:49.970930389 +0000 UTC m=+217.514236148" Sep 16 18:48:50.035609 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:50.035577 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-j4v26_19beda34-68cc-4379-8bf4-7d41f797995e/monitoring-plugin/0.log" Sep 16 18:48:50.363557 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:50.363511 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:50.363730 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:50.363575 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:50.368740 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:50.368714 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:50.945311 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:50.945282 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:48:50.998593 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:50.998544 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-668d8b7689-bqk9w"] Sep 16 18:48:51.436096 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:51.436063 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwd8j_ee5fee63-a1d4-432a-928b-5946e3253bf4/init-textfile/0.log" Sep 16 18:48:51.637250 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:51.637218 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwd8j_ee5fee63-a1d4-432a-928b-5946e3253bf4/node-exporter/0.log" Sep 16 18:48:51.836619 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:51.836580 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwd8j_ee5fee63-a1d4-432a-928b-5946e3253bf4/kube-rbac-proxy/0.log" Sep 16 18:48:52.038530 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:52.038495 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sjmf8_798f8fb8-07f0-4c07-84ff-56d9abb88cd9/kube-rbac-proxy-main/0.log" Sep 16 18:48:52.236652 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:52.236619 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sjmf8_798f8fb8-07f0-4c07-84ff-56d9abb88cd9/kube-rbac-proxy-self/0.log" Sep 16 18:48:52.436514 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:52.436484 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sjmf8_798f8fb8-07f0-4c07-84ff-56d9abb88cd9/openshift-state-metrics/0.log" Sep 16 18:48:54.436886 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:54.436837 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-lqblt_ba9411be-e65f-428e-86ef-fd4284a981cf/prometheus-operator-admission-webhook/0.log" Sep 16 18:48:56.436935 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:56.436902 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-668d8b7689-bqk9w_7ee1dc8b-8d6c-45d7-94d2-744f9b4016be/console/0.log" Sep 16 18:48:56.635851 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:56.635815 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-947d58874-rtnpp_823760f5-fcef-47c4-a1cf-a35209fba9fc/console/0.log" Sep 16 18:48:56.837911 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:56.837883 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-qxsx2_feed169e-b4a9-476f-82a3-250cc36f122e/download-server/0.log" Sep 16 18:48:58.148261 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:58.148232 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:48:59.166668 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:59.166637 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:48:59.171145 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:48:59.171118 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69dd66586b-btdkd" Sep 16 18:49:16.968046 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:16.968008 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-668d8b7689-bqk9w" podUID="7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" containerName="console" containerID="cri-o://5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d" gracePeriod=15 Sep 16 18:49:17.252843 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.252807 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:49:17.406943 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.406903 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-service-ca\") pod \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " Sep 16 18:49:17.406943 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.406944 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-oauth-serving-cert\") pod \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " Sep 16 18:49:17.407149 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.406984 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-config\") pod \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " Sep 16 18:49:17.407149 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407003 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8lbl\" (UniqueName: \"kubernetes.io/projected/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-kube-api-access-v8lbl\") pod \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " Sep 16 18:49:17.407232 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407196 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-serving-cert\") pod \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " Sep 16 18:49:17.407309 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407291 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-oauth-config\") pod \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\" (UID: \"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be\") " Sep 16 18:49:17.407433 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407404 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-service-ca" (OuterVolumeSpecName: "service-ca") pod "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" (UID: "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:17.407433 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407410 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" (UID: "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:17.407433 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407426 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-config" (OuterVolumeSpecName: "console-config") pod "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" (UID: "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:17.407655 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407560 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-service-ca\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:17.407655 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407585 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-oauth-serving-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:17.407655 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.407602 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:17.409539 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.409509 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" (UID: "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:17.409710 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.409551 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" (UID: "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:17.409710 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.409560 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-kube-api-access-v8lbl" (OuterVolumeSpecName: "kube-api-access-v8lbl") pod "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" (UID: "7ee1dc8b-8d6c-45d7-94d2-744f9b4016be"). InnerVolumeSpecName "kube-api-access-v8lbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:49:17.508322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.508235 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8lbl\" (UniqueName: \"kubernetes.io/projected/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-kube-api-access-v8lbl\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:17.508322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.508267 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-serving-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:17.508322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:17.508278 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be-console-oauth-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:18.052753 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.052718 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" containerID="5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d" exitCode=0 Sep 16 18:49:18.053211 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.052767 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-668d8b7689-bqk9w" Sep 16 18:49:18.053211 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.052789 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668d8b7689-bqk9w" event={"ID":"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be","Type":"ContainerDied","Data":"5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d"} Sep 16 18:49:18.053211 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.052839 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-668d8b7689-bqk9w" event={"ID":"7ee1dc8b-8d6c-45d7-94d2-744f9b4016be","Type":"ContainerDied","Data":"abc57e6ac967a9be522762daec18dca1f478c1fd97852657f48bf7ac8f97fee9"} Sep 16 18:49:18.053211 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.052890 2570 scope.go:117] "RemoveContainer" containerID="5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d" Sep 16 18:49:18.061548 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.061529 2570 scope.go:117] "RemoveContainer" containerID="5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d" Sep 16 18:49:18.061841 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:49:18.061820 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d\": container with ID starting with 5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d not found: ID does not exist" containerID="5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d" Sep 16 18:49:18.061899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.061851 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d"} err="failed to get container status \"5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d\": rpc error: code = NotFound desc = could not find container \"5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d\": container with ID starting with 5e6dbc19fa9b65cd2eb95312499f4b647ac9e7e8f1046c3233342bd6d80ee34d not found: ID does not exist" Sep 16 18:49:18.074130 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.074106 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-668d8b7689-bqk9w"] Sep 16 18:49:18.077591 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.077570 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-668d8b7689-bqk9w"] Sep 16 18:49:18.970588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:18.970555 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" path="/var/lib/kubelet/pods/7ee1dc8b-8d6c-45d7-94d2-744f9b4016be/volumes" Sep 16 18:49:24.876131 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:24.876091 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:49:24.878546 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:24.878516 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba-metrics-certs\") pod \"network-metrics-daemon-mllt4\" (UID: \"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba\") " pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:49:25.168704 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:25.168621 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7n2q\"" Sep 16 18:49:25.177468 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:25.177444 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-mllt4" Sep 16 18:49:25.481842 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:25.481814 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-mllt4"] Sep 16 18:49:25.485039 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:49:25.484999 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f7e4b9_8c62_41f6_a0c9_71f149faa2ba.slice/crio-518afd90cce20ddac13eb0fc971f7df67b8341c823dde671e969f54d6029a2b3 WatchSource:0}: Error finding container 518afd90cce20ddac13eb0fc971f7df67b8341c823dde671e969f54d6029a2b3: Status 404 returned error can't find the container with id 518afd90cce20ddac13eb0fc971f7df67b8341c823dde671e969f54d6029a2b3 Sep 16 18:49:26.088810 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:26.088778 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mllt4" event={"ID":"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba","Type":"ContainerStarted","Data":"518afd90cce20ddac13eb0fc971f7df67b8341c823dde671e969f54d6029a2b3"} Sep 16 18:49:27.094562 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:27.094525 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mllt4" event={"ID":"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba","Type":"ContainerStarted","Data":"cb0af6bde2c9e0b942ecb359345b8f14bbd41cedb1f0aa770344f448640fe91d"} Sep 16 18:49:27.095052 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:27.094582 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-mllt4" event={"ID":"f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba","Type":"ContainerStarted","Data":"c530c6b5db1cada490068d6edc673980b13ac28624d7ed8e37a77a527c3c53b0"} Sep 16 18:49:27.112799 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:27.112751 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-mllt4" podStartSLOduration=252.880138677 podStartE2EDuration="4m14.112734985s" totalImagesPullingTime="1.232596308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:27.1116288 +0000 UTC m=+254.654934549" watchObservedRunningTime="2026-09-16 18:49:27.112734985 +0000 UTC m=+254.656040744" Sep 16 18:49:34.971052 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:34.970975 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:49:35.125432 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:35.125387 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="alertmanager" containerID="cri-o://66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57" gracePeriod=120 Sep 16 18:49:35.125432 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:35.125433 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy-metric" containerID="cri-o://fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab" gracePeriod=120 Sep 16 18:49:35.125703 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:35.125482 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy-web" containerID="cri-o://06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4" gracePeriod=120 Sep 16 18:49:35.125703 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:35.125494 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy" containerID="cri-o://ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117" gracePeriod=120 Sep 16 18:49:35.125703 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:35.125534 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="prom-label-proxy" containerID="cri-o://3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857" gracePeriod=120 Sep 16 18:49:35.125703 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:35.125499 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="config-reloader" containerID="cri-o://c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133" gracePeriod=120 Sep 16 18:49:36.132416 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.132383 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerID="3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857" exitCode=0 Sep 16 18:49:36.132416 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.132406 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerID="ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117" exitCode=0 Sep 16 18:49:36.132416 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.132412 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerID="c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133" exitCode=0 Sep 16 18:49:36.132416 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.132418 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerID="66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57" exitCode=0 Sep 16 18:49:36.132888 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.132451 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerDied","Data":"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857"} Sep 16 18:49:36.132888 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.132516 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerDied","Data":"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117"} Sep 16 18:49:36.132888 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.132544 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerDied","Data":"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133"} Sep 16 18:49:36.132888 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.132568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerDied","Data":"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57"} Sep 16 18:49:36.377457 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.377432 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:36.451986 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.451909 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.451986 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.451947 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.451986 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.451965 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-web\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.451986 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.451984 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-out\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.451999 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-trusted-ca-bundle\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452027 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-metrics-client-ca\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452050 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n29jt\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-kube-api-access-n29jt\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452066 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-volume\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452098 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-tls-assets\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452128 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-web-config\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452180 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-main-db\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452211 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-main-tls\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452301 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452236 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-cluster-tls-config\") pod \"3f3b8782-c5b8-4cb5-b635-043b63fad944\" (UID: \"3f3b8782-c5b8-4cb5-b635-043b63fad944\") " Sep 16 18:49:36.452734 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.452645 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:36.453606 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.453185 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:49:36.453606 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.453335 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:36.456420 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.456374 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:36.456609 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.456506 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:36.456834 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.456799 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-out" (OuterVolumeSpecName: "config-out") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:49:36.456937 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.456847 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:36.456937 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.456851 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:49:36.457147 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.457118 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-volume" (OuterVolumeSpecName: "config-volume") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:36.457333 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.457307 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:36.458339 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.458309 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-kube-api-access-n29jt" (OuterVolumeSpecName: "kube-api-access-n29jt") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "kube-api-access-n29jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:49:36.462659 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.462637 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:36.467509 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.467487 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-web-config" (OuterVolumeSpecName: "web-config") pod "3f3b8782-c5b8-4cb5-b635-043b63fad944" (UID: "3f3b8782-c5b8-4cb5-b635-043b63fad944"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:36.552779 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552744 2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-main-db\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.552779 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552776 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-main-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.552779 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552789 2570 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-cluster-tls-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552802 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552817 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552830 2570 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552841 2570 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-out\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552852 2570 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552890 2570 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b8782-c5b8-4cb5-b635-043b63fad944-metrics-client-ca\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552903 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n29jt\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-kube-api-access-n29jt\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552913 2570 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-config-volume\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552926 2570 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3f3b8782-c5b8-4cb5-b635-043b63fad944-tls-assets\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:36.553074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:36.552938 2570 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3f3b8782-c5b8-4cb5-b635-043b63fad944-web-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:49:37.139702 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.139667 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerID="fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab" exitCode=0 Sep 16 18:49:37.139702 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.139690 2570 generic.go:356] "Generic (PLEG): container finished" podID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerID="06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4" exitCode=0 Sep 16 18:49:37.140234 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.139744 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerDied","Data":"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab"} Sep 16 18:49:37.140234 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.139758 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.140234 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.139817 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerDied","Data":"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4"} Sep 16 18:49:37.140234 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.139849 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3f3b8782-c5b8-4cb5-b635-043b63fad944","Type":"ContainerDied","Data":"c86f4904e84ef63a6cd2980317c15814369e14f2bd12a314b5895a54449fa791"} Sep 16 18:49:37.140234 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.139885 2570 scope.go:117] "RemoveContainer" containerID="3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857" Sep 16 18:49:37.149639 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.149619 2570 scope.go:117] "RemoveContainer" containerID="fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab" Sep 16 18:49:37.156912 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.156888 2570 scope.go:117] "RemoveContainer" containerID="ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117" Sep 16 18:49:37.160877 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.160837 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:49:37.164958 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.164896 2570 scope.go:117] "RemoveContainer" containerID="06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4" Sep 16 18:49:37.168553 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.168526 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:49:37.176618 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.176594 2570 scope.go:117] "RemoveContainer" containerID="c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133" Sep 16 18:49:37.184195 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.184178 2570 scope.go:117] "RemoveContainer" containerID="66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57" Sep 16 18:49:37.191448 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.191428 2570 scope.go:117] "RemoveContainer" containerID="ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c" Sep 16 18:49:37.196760 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.196738 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197119 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="config-reloader" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197135 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="config-reloader" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197144 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="init-config-reloader" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197150 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="init-config-reloader" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197163 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="prom-label-proxy" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197169 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="prom-label-proxy" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197175 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" containerName="console" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197180 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" containerName="console" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197185 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy-web" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197190 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy-web" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197197 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy-metric" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197202 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy-metric" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197211 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197216 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197223 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="alertmanager" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197228 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="alertmanager" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197275 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy-web" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197281 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy-metric" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197288 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="config-reloader" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197296 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="alertmanager" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197302 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="kube-rbac-proxy" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197307 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" containerName="prom-label-proxy" Sep 16 18:49:37.197365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.197312 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ee1dc8b-8d6c-45d7-94d2-744f9b4016be" containerName="console" Sep 16 18:49:37.201222 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.201208 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.203721 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.203699 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kd2hz\"" Sep 16 18:49:37.203893 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.203877 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 16 18:49:37.204022 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.203884 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 16 18:49:37.204155 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.204138 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 16 18:49:37.204258 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.204152 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 16 18:49:37.204258 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.204254 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 16 18:49:37.204478 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.204457 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 16 18:49:37.204577 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.204554 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 16 18:49:37.204701 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.204683 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 16 18:49:37.205452 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.205438 2570 scope.go:117] "RemoveContainer" containerID="3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857" Sep 16 18:49:37.205905 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:49:37.205851 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857\": container with ID starting with 3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857 not found: ID does not exist" containerID="3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857" Sep 16 18:49:37.205998 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.205913 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857"} err="failed to get container status \"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857\": rpc error: code = NotFound desc = could not find container \"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857\": container with ID starting with 3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857 not found: ID does not exist" Sep 16 18:49:37.205998 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.205931 2570 scope.go:117] "RemoveContainer" containerID="fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab" Sep 16 18:49:37.206217 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:49:37.206200 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab\": container with ID starting with fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab not found: ID does not exist" containerID="fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab" Sep 16 18:49:37.206267 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.206225 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab"} err="failed to get container status \"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab\": rpc error: code = NotFound desc = could not find container \"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab\": container with ID starting with fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab not found: ID does not exist" Sep 16 18:49:37.206267 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.206245 2570 scope.go:117] "RemoveContainer" containerID="ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117" Sep 16 18:49:37.206550 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:49:37.206531 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117\": container with ID starting with ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117 not found: ID does not exist" containerID="ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117" Sep 16 18:49:37.206607 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.206556 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117"} err="failed to get container status \"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117\": rpc error: code = NotFound desc = could not find container \"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117\": container with ID starting with ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117 not found: ID does not exist" Sep 16 18:49:37.206607 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.206568 2570 scope.go:117] "RemoveContainer" containerID="06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4" Sep 16 18:49:37.206884 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:49:37.206845 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4\": container with ID starting with 06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4 not found: ID does not exist" containerID="06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4" Sep 16 18:49:37.206939 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.206891 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4"} err="failed to get container status \"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4\": rpc error: code = NotFound desc = could not find container \"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4\": container with ID starting with 06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4 not found: ID does not exist" Sep 16 18:49:37.206939 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.206913 2570 scope.go:117] "RemoveContainer" containerID="c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133" Sep 16 18:49:37.207249 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:49:37.207231 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133\": container with ID starting with c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133 not found: ID does not exist" containerID="c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133" Sep 16 18:49:37.207343 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.207252 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133"} err="failed to get container status \"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133\": rpc error: code = NotFound desc = could not find container \"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133\": container with ID starting with c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133 not found: ID does not exist" Sep 16 18:49:37.207343 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.207265 2570 scope.go:117] "RemoveContainer" containerID="66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57" Sep 16 18:49:37.207606 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:49:37.207586 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57\": container with ID starting with 66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57 not found: ID does not exist" containerID="66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57" Sep 16 18:49:37.207748 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.207618 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57"} err="failed to get container status \"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57\": rpc error: code = NotFound desc = could not find container \"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57\": container with ID starting with 66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57 not found: ID does not exist" Sep 16 18:49:37.207748 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.207633 2570 scope.go:117] "RemoveContainer" containerID="ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c" Sep 16 18:49:37.207972 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:49:37.207952 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c\": container with ID starting with ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c not found: ID does not exist" containerID="ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c" Sep 16 18:49:37.208056 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.207975 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c"} err="failed to get container status \"ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c\": rpc error: code = NotFound desc = could not find container \"ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c\": container with ID starting with ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c not found: ID does not exist" Sep 16 18:49:37.208056 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.207990 2570 scope.go:117] "RemoveContainer" containerID="3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857" Sep 16 18:49:37.208278 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.208256 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857"} err="failed to get container status \"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857\": rpc error: code = NotFound desc = could not find container \"3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857\": container with ID starting with 3ce13a2bc86f4b6f7949255d2b7a295fb6a139a337b94349c2a1c642ea578857 not found: ID does not exist" Sep 16 18:49:37.208354 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.208322 2570 scope.go:117] "RemoveContainer" containerID="fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab" Sep 16 18:49:37.208604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.208586 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab"} err="failed to get container status \"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab\": rpc error: code = NotFound desc = could not find container \"fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab\": container with ID starting with fad5aaa41dff03d3d2eb837dba8130a6e5f9520c5b7922ce7a2a655a2956e4ab not found: ID does not exist" Sep 16 18:49:37.208677 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.208606 2570 scope.go:117] "RemoveContainer" containerID="ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117" Sep 16 18:49:37.209092 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.209064 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117"} err="failed to get container status \"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117\": rpc error: code = NotFound desc = could not find container \"ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117\": container with ID starting with ab3a171ff5f68f5a1a80247cb33ca5fc989f79559518e830dc24e3d5016a1117 not found: ID does not exist" Sep 16 18:49:37.209179 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.209115 2570 scope.go:117] "RemoveContainer" containerID="06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4" Sep 16 18:49:37.209407 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.209390 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4"} err="failed to get container status \"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4\": rpc error: code = NotFound desc = could not find container \"06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4\": container with ID starting with 06691a4b578a0d4149519919cefa80ed9f9719f08cb8c4d474c68e906a4105b4 not found: ID does not exist" Sep 16 18:49:37.209473 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.209408 2570 scope.go:117] "RemoveContainer" containerID="c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133" Sep 16 18:49:37.209672 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.209655 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133"} err="failed to get container status \"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133\": rpc error: code = NotFound desc = could not find container \"c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133\": container with ID starting with c233ef80d0d9a10850daf18fa8e67621d4068100d4a026587f7849106b9e3133 not found: ID does not exist" Sep 16 18:49:37.209727 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.209673 2570 scope.go:117] "RemoveContainer" containerID="66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57" Sep 16 18:49:37.209967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.209925 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57"} err="failed to get container status \"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57\": rpc error: code = NotFound desc = could not find container \"66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57\": container with ID starting with 66c99d1bf2a872b235ee216cb1d7df3cb078a23f0521ed0edcf34f2939dbcd57 not found: ID does not exist" Sep 16 18:49:37.209967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.209958 2570 scope.go:117] "RemoveContainer" containerID="ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c" Sep 16 18:49:37.210240 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.210222 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c"} err="failed to get container status \"ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c\": rpc error: code = NotFound desc = could not find container \"ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c\": container with ID starting with ab0ae9d756d019b4a51716d40c952b5379926fb6d140432e7babb9cfda17529c not found: ID does not exist" Sep 16 18:49:37.210298 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.210250 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 16 18:49:37.221255 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.221228 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:49:37.257992 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.257918 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.257992 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.257949 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.257992 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.257967 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.257992 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.257982 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.257992 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.257998 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-web-config\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.258263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.258059 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-config-out\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.258263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.258092 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.258263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.258155 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.258263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.258182 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.258263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.258206 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6s6h\" (UniqueName: \"kubernetes.io/projected/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-kube-api-access-m6s6h\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.258263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.258248 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.258441 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.258277 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.258441 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.258346 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359151 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359116 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6s6h\" (UniqueName: \"kubernetes.io/projected/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-kube-api-access-m6s6h\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359180 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359201 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359222 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359263 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359254 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359493 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359272 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359493 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359290 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359493 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359307 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359493 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359329 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-web-config\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.359747 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.359721 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.361532 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.360427 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.361532 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.360492 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-config-out\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.361532 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.360525 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.361532 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.360585 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.361532 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.361243 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.362530 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.362502 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.362639 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.362502 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-web-config\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.362882 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.362845 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.362967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.362848 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.362967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.362848 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.363353 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.363331 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.363426 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.363363 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.363977 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.363959 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-config-out\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.364729 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.364706 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.368225 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.368209 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6s6h\" (UniqueName: \"kubernetes.io/projected/5c8ef80e-b4d9-4c76-b595-24e84ce6730a-kube-api-access-m6s6h\") pod \"alertmanager-main-0\" (UID: \"5c8ef80e-b4d9-4c76-b595-24e84ce6730a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.512099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.512012 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 16 18:49:37.794218 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:37.794123 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 16 18:49:37.798533 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:49:37.798496 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8ef80e_b4d9_4c76_b595_24e84ce6730a.slice/crio-3d1624da46cdb1026018ef574da096df186a3fe1af63e384b2cd8bbc401ca378 WatchSource:0}: Error finding container 3d1624da46cdb1026018ef574da096df186a3fe1af63e384b2cd8bbc401ca378: Status 404 returned error can't find the container with id 3d1624da46cdb1026018ef574da096df186a3fe1af63e384b2cd8bbc401ca378 Sep 16 18:49:38.145985 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:38.145882 2570 generic.go:356] "Generic (PLEG): container finished" podID="5c8ef80e-b4d9-4c76-b595-24e84ce6730a" containerID="8ac9a081c0c99440638fafc62b9263b2ad4d33f3ec20878a3eb9c4da21765bde" exitCode=0 Sep 16 18:49:38.145985 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:38.145956 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c8ef80e-b4d9-4c76-b595-24e84ce6730a","Type":"ContainerDied","Data":"8ac9a081c0c99440638fafc62b9263b2ad4d33f3ec20878a3eb9c4da21765bde"} Sep 16 18:49:38.146496 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:38.146029 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c8ef80e-b4d9-4c76-b595-24e84ce6730a","Type":"ContainerStarted","Data":"3d1624da46cdb1026018ef574da096df186a3fe1af63e384b2cd8bbc401ca378"} Sep 16 18:49:38.971686 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:38.971652 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3b8782-c5b8-4cb5-b635-043b63fad944" path="/var/lib/kubelet/pods/3f3b8782-c5b8-4cb5-b635-043b63fad944/volumes" Sep 16 18:49:39.033878 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.033843 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-76d5f6d88b-t22cc"] Sep 16 18:49:39.036288 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.036272 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.038753 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.038733 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 16 18:49:39.038996 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.038979 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zvt88\"" Sep 16 18:49:39.039073 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.038984 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 16 18:49:39.039073 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.039041 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 16 18:49:39.039348 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.039329 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 16 18:49:39.039423 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.039342 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 16 18:49:39.048367 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.046036 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 16 18:49:39.053106 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.053079 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76d5f6d88b-t22cc"] Sep 16 18:49:39.073779 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.073739 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-telemeter-client-tls\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.073779 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.073780 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-serving-certs-ca-bundle\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.074062 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.073823 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-federate-client-tls\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.074062 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.073844 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-secret-telemeter-client\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.074062 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.073889 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmdq\" (UniqueName: \"kubernetes.io/projected/5c022665-6622-4270-b4ea-7406efdead2d-kube-api-access-lxmdq\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.074062 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.073916 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-metrics-client-ca\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.074062 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.073940 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.074062 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.073987 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.153499 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.153464 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c8ef80e-b4d9-4c76-b595-24e84ce6730a","Type":"ContainerStarted","Data":"ae5e8adcc98856773a6fb8792998c41494f9c724fffe3021eaf659fff2bc5f68"} Sep 16 18:49:39.153995 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.153542 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c8ef80e-b4d9-4c76-b595-24e84ce6730a","Type":"ContainerStarted","Data":"f05a85264418c505eb24c0f1fd90eda29844e0459326adf158c0aa9212c039e8"} Sep 16 18:49:39.153995 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.153580 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c8ef80e-b4d9-4c76-b595-24e84ce6730a","Type":"ContainerStarted","Data":"eb16f2c249a2a6fe08a4fa2e46caf06d896f7a1e289134feef3e5ffadf3d969b"} Sep 16 18:49:39.153995 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.153630 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c8ef80e-b4d9-4c76-b595-24e84ce6730a","Type":"ContainerStarted","Data":"5276b54a9a0b0478e104700815eacf8ca6c66fb7cd69d5f4cb109eac4b65670d"} Sep 16 18:49:39.153995 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.153670 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c8ef80e-b4d9-4c76-b595-24e84ce6730a","Type":"ContainerStarted","Data":"5f6018e2bf950076287adb77e9eceb68f67e7a1d95448b9e84f4cdbd5e38f6e4"} Sep 16 18:49:39.153995 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.153708 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5c8ef80e-b4d9-4c76-b595-24e84ce6730a","Type":"ContainerStarted","Data":"b3449cbbdc8799cb6b3258d220535816fa3ad919568ef3f161c3d7ca1ea17ad8"} Sep 16 18:49:39.174391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.174357 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-federate-client-tls\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.174391 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.174392 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-secret-telemeter-client\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.174631 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.174444 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmdq\" (UniqueName: \"kubernetes.io/projected/5c022665-6622-4270-b4ea-7406efdead2d-kube-api-access-lxmdq\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.174631 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.174501 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-metrics-client-ca\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.174739 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.174671 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.174857 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.174833 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.174964 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.174940 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-telemeter-client-tls\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.175026 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.175003 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-serving-certs-ca-bundle\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.175650 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.175258 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-metrics-client-ca\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.175808 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.175785 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-serving-certs-ca-bundle\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.175937 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.175914 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c022665-6622-4270-b4ea-7406efdead2d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.177412 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.177385 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.177565 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.177548 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-telemeter-client-tls\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.177615 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.177591 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-federate-client-tls\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.177658 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.177623 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/5c022665-6622-4270-b4ea-7406efdead2d-secret-telemeter-client\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.181764 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.181722 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.167463604 podStartE2EDuration="2.181707391s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.243787ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:39.17969864 +0000 UTC m=+266.723004403" watchObservedRunningTime="2026-09-16 18:49:39.181707391 +0000 UTC m=+266.725013149" Sep 16 18:49:39.184471 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.184454 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmdq\" (UniqueName: \"kubernetes.io/projected/5c022665-6622-4270-b4ea-7406efdead2d-kube-api-access-lxmdq\") pod \"telemeter-client-76d5f6d88b-t22cc\" (UID: \"5c022665-6622-4270-b4ea-7406efdead2d\") " pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.350733 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.350625 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" Sep 16 18:49:39.634604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:39.634578 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-76d5f6d88b-t22cc"] Sep 16 18:49:39.636928 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:49:39.636901 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c022665_6622_4270_b4ea_7406efdead2d.slice/crio-40d7d4f7c295fc038a137b109ab61a084ed6a4c55b65b09c151736997c885fbb WatchSource:0}: Error finding container 40d7d4f7c295fc038a137b109ab61a084ed6a4c55b65b09c151736997c885fbb: Status 404 returned error can't find the container with id 40d7d4f7c295fc038a137b109ab61a084ed6a4c55b65b09c151736997c885fbb Sep 16 18:49:40.160847 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:40.160807 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" event={"ID":"5c022665-6622-4270-b4ea-7406efdead2d","Type":"ContainerStarted","Data":"40d7d4f7c295fc038a137b109ab61a084ed6a4c55b65b09c151736997c885fbb"} Sep 16 18:49:42.173588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.173497 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" event={"ID":"5c022665-6622-4270-b4ea-7406efdead2d","Type":"ContainerStarted","Data":"15e05176dbb55425f0bb22a9a7a1dae12c112ce54f91e62540b289570f183bed"} Sep 16 18:49:42.173588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.173549 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" event={"ID":"5c022665-6622-4270-b4ea-7406efdead2d","Type":"ContainerStarted","Data":"3e0a9a6ed65a7c8ca1a40ea8bae32e58c116480db96bf08d3330687aa5505d86"} Sep 16 18:49:42.173588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.173573 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" event={"ID":"5c022665-6622-4270-b4ea-7406efdead2d","Type":"ContainerStarted","Data":"88991427e801dd9dcde30556354b05c54e7a087c03ade7b40156c6612c5fb667"} Sep 16 18:49:42.194177 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.194126 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-76d5f6d88b-t22cc" podStartSLOduration=1.027603032 podStartE2EDuration="3.194100862s" totalImagesPullingTime="2.16649783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:42.193114985 +0000 UTC m=+269.736420745" watchObservedRunningTime="2026-09-16 18:49:42.194100862 +0000 UTC m=+269.737406620" Sep 16 18:49:42.270710 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.270668 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-554c748f5d-6n2mn"] Sep 16 18:49:42.275067 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.275029 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.281432 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.281399 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554c748f5d-6n2mn"] Sep 16 18:49:42.300245 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.300209 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9w52\" (UniqueName: \"kubernetes.io/projected/8f73caf1-9560-4516-a5fc-b805717168f8-kube-api-access-g9w52\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.300426 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.300271 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-oauth-config\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.300426 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.300339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-oauth-serving-cert\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.300426 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.300365 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-serving-cert\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.300426 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.300386 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-console-config\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.300426 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.300402 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-trusted-ca-bundle\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.300426 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.300419 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-service-ca\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.400843 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.400802 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-oauth-config\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.401068 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.400899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-oauth-serving-cert\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.401068 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.400936 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-serving-cert\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.401068 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.400959 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-console-config\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.401227 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.401103 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-trusted-ca-bundle\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.401227 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.401156 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-service-ca\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.401227 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.401198 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9w52\" (UniqueName: \"kubernetes.io/projected/8f73caf1-9560-4516-a5fc-b805717168f8-kube-api-access-g9w52\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.401987 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.401960 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-console-config\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.402118 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.401989 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-oauth-serving-cert\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.402118 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.402012 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-trusted-ca-bundle\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.402118 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.402020 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-service-ca\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.403688 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.403666 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-serving-cert\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.403778 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.403691 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-oauth-config\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.409910 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.409857 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9w52\" (UniqueName: \"kubernetes.io/projected/8f73caf1-9560-4516-a5fc-b805717168f8-kube-api-access-g9w52\") pod \"console-554c748f5d-6n2mn\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.586026 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.585981 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:42.860560 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:42.860363 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554c748f5d-6n2mn"] Sep 16 18:49:42.862875 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:49:42.862843 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f73caf1_9560_4516_a5fc_b805717168f8.slice/crio-9bbf5dbff888bb557a84a4814cf1b908669985295f5774959837bee7fcfe849f WatchSource:0}: Error finding container 9bbf5dbff888bb557a84a4814cf1b908669985295f5774959837bee7fcfe849f: Status 404 returned error can't find the container with id 9bbf5dbff888bb557a84a4814cf1b908669985295f5774959837bee7fcfe849f Sep 16 18:49:43.180164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:43.180070 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554c748f5d-6n2mn" event={"ID":"8f73caf1-9560-4516-a5fc-b805717168f8","Type":"ContainerStarted","Data":"cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3"} Sep 16 18:49:43.180164 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:43.180152 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554c748f5d-6n2mn" event={"ID":"8f73caf1-9560-4516-a5fc-b805717168f8","Type":"ContainerStarted","Data":"9bbf5dbff888bb557a84a4814cf1b908669985295f5774959837bee7fcfe849f"} Sep 16 18:49:43.199566 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:43.199514 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-554c748f5d-6n2mn" podStartSLOduration=1.199496167 podStartE2EDuration="1.199496167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:43.197842691 +0000 UTC m=+270.741148472" watchObservedRunningTime="2026-09-16 18:49:43.199496167 +0000 UTC m=+270.742801926" Sep 16 18:49:52.587031 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:52.586968 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:52.587491 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:52.587077 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:52.591831 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:52.591803 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:53.233064 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:53.233032 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:49:53.285333 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:49:53.285295 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-947d58874-rtnpp"] Sep 16 18:50:12.868099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:12.868070 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:50:12.874604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:12.874318 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:50:12.876682 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:12.876662 2570 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 18:50:18.306224 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.306156 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-947d58874-rtnpp" podUID="823760f5-fcef-47c4-a1cf-a35209fba9fc" containerName="console" containerID="cri-o://519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12" gracePeriod=15 Sep 16 18:50:18.550466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.550437 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:50:18.714618 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.714584 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-oauth-serving-cert\") pod \"823760f5-fcef-47c4-a1cf-a35209fba9fc\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " Sep 16 18:50:18.714820 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.714639 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-oauth-config\") pod \"823760f5-fcef-47c4-a1cf-a35209fba9fc\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " Sep 16 18:50:18.714820 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.714664 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmj4v\" (UniqueName: \"kubernetes.io/projected/823760f5-fcef-47c4-a1cf-a35209fba9fc-kube-api-access-kmj4v\") pod \"823760f5-fcef-47c4-a1cf-a35209fba9fc\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " Sep 16 18:50:18.714820 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.714701 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-config\") pod \"823760f5-fcef-47c4-a1cf-a35209fba9fc\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " Sep 16 18:50:18.714820 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.714736 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-service-ca\") pod \"823760f5-fcef-47c4-a1cf-a35209fba9fc\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " Sep 16 18:50:18.715095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.714829 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-serving-cert\") pod \"823760f5-fcef-47c4-a1cf-a35209fba9fc\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " Sep 16 18:50:18.715095 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.714853 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-trusted-ca-bundle\") pod \"823760f5-fcef-47c4-a1cf-a35209fba9fc\" (UID: \"823760f5-fcef-47c4-a1cf-a35209fba9fc\") " Sep 16 18:50:18.715188 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.715113 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "823760f5-fcef-47c4-a1cf-a35209fba9fc" (UID: "823760f5-fcef-47c4-a1cf-a35209fba9fc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:50:18.715188 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.715135 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-config" (OuterVolumeSpecName: "console-config") pod "823760f5-fcef-47c4-a1cf-a35209fba9fc" (UID: "823760f5-fcef-47c4-a1cf-a35209fba9fc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:50:18.715569 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.715403 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "823760f5-fcef-47c4-a1cf-a35209fba9fc" (UID: "823760f5-fcef-47c4-a1cf-a35209fba9fc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:50:18.715716 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.715688 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-service-ca" (OuterVolumeSpecName: "service-ca") pod "823760f5-fcef-47c4-a1cf-a35209fba9fc" (UID: "823760f5-fcef-47c4-a1cf-a35209fba9fc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:50:18.717092 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.717055 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "823760f5-fcef-47c4-a1cf-a35209fba9fc" (UID: "823760f5-fcef-47c4-a1cf-a35209fba9fc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:50:18.717196 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.717101 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823760f5-fcef-47c4-a1cf-a35209fba9fc-kube-api-access-kmj4v" (OuterVolumeSpecName: "kube-api-access-kmj4v") pod "823760f5-fcef-47c4-a1cf-a35209fba9fc" (UID: "823760f5-fcef-47c4-a1cf-a35209fba9fc"). InnerVolumeSpecName "kube-api-access-kmj4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:50:18.717196 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.717103 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "823760f5-fcef-47c4-a1cf-a35209fba9fc" (UID: "823760f5-fcef-47c4-a1cf-a35209fba9fc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:50:18.816473 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.816433 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-service-ca\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:50:18.816473 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.816466 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-serving-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:50:18.816473 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.816476 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-trusted-ca-bundle\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:50:18.816670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.816485 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-oauth-serving-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:50:18.816670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.816493 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-oauth-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:50:18.816670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.816502 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmj4v\" (UniqueName: \"kubernetes.io/projected/823760f5-fcef-47c4-a1cf-a35209fba9fc-kube-api-access-kmj4v\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:50:18.816670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:18.816510 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/823760f5-fcef-47c4-a1cf-a35209fba9fc-console-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:50:19.349101 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.349005 2570 generic.go:356] "Generic (PLEG): container finished" podID="823760f5-fcef-47c4-a1cf-a35209fba9fc" containerID="519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12" exitCode=0 Sep 16 18:50:19.349101 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.349048 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-947d58874-rtnpp" Sep 16 18:50:19.349101 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.349086 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-947d58874-rtnpp" event={"ID":"823760f5-fcef-47c4-a1cf-a35209fba9fc","Type":"ContainerDied","Data":"519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12"} Sep 16 18:50:19.349623 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.349137 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-947d58874-rtnpp" event={"ID":"823760f5-fcef-47c4-a1cf-a35209fba9fc","Type":"ContainerDied","Data":"f3e7cfcc6c40d66597d34e5e62ccc3aa9ba8ba7db2d7a681bdcaeef93de68e3e"} Sep 16 18:50:19.349623 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.349158 2570 scope.go:117] "RemoveContainer" containerID="519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12" Sep 16 18:50:19.357541 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.357523 2570 scope.go:117] "RemoveContainer" containerID="519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12" Sep 16 18:50:19.357828 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:50:19.357806 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12\": container with ID starting with 519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12 not found: ID does not exist" containerID="519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12" Sep 16 18:50:19.357920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.357842 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12"} err="failed to get container status \"519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12\": rpc error: code = NotFound desc = could not find container \"519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12\": container with ID starting with 519c75a37c2befc19dd622eb8fe8c1798b3e6c14edd31ee40dd2f03dba871a12 not found: ID does not exist" Sep 16 18:50:19.365621 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.365593 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-947d58874-rtnpp"] Sep 16 18:50:19.369657 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:19.369634 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-947d58874-rtnpp"] Sep 16 18:50:20.970298 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:20.970266 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="823760f5-fcef-47c4-a1cf-a35209fba9fc" path="/var/lib/kubelet/pods/823760f5-fcef-47c4-a1cf-a35209fba9fc/volumes" Sep 16 18:50:50.428916 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.428879 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-547d7b6b54-vgl9d"] Sep 16 18:50:50.429423 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.429254 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="823760f5-fcef-47c4-a1cf-a35209fba9fc" containerName="console" Sep 16 18:50:50.429423 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.429269 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="823760f5-fcef-47c4-a1cf-a35209fba9fc" containerName="console" Sep 16 18:50:50.429423 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.429338 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="823760f5-fcef-47c4-a1cf-a35209fba9fc" containerName="console" Sep 16 18:50:50.432710 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.432694 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.446018 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.445985 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-547d7b6b54-vgl9d"] Sep 16 18:50:50.491464 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.491418 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pppl7\" (UniqueName: \"kubernetes.io/projected/adfd869f-6302-4b33-8128-03c52c0dccef-kube-api-access-pppl7\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.491464 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.491457 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-oauth-config\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.491682 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.491511 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-console-config\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.491682 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.491533 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-trusted-ca-bundle\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.491682 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.491554 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-serving-cert\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.491682 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.491577 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-service-ca\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.491682 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.491648 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-oauth-serving-cert\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.592406 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.592376 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-console-config\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.592570 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.592420 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-trusted-ca-bundle\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.592570 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.592457 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-serving-cert\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.592570 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.592489 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-service-ca\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.592570 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.592522 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-oauth-serving-cert\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.592570 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.592563 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pppl7\" (UniqueName: \"kubernetes.io/projected/adfd869f-6302-4b33-8128-03c52c0dccef-kube-api-access-pppl7\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.592813 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.592592 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-oauth-config\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.593242 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.593214 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-console-config\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.593363 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.593283 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-service-ca\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.593363 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.593315 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-oauth-serving-cert\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.593363 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.593345 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-trusted-ca-bundle\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.595193 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.595173 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-oauth-config\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.595298 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.595283 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-serving-cert\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.602121 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.602093 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pppl7\" (UniqueName: \"kubernetes.io/projected/adfd869f-6302-4b33-8128-03c52c0dccef-kube-api-access-pppl7\") pod \"console-547d7b6b54-vgl9d\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:50.742162 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:50.742127 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:50:51.018424 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:51.018316 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-547d7b6b54-vgl9d"] Sep 16 18:50:51.479625 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:51.479590 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547d7b6b54-vgl9d" event={"ID":"adfd869f-6302-4b33-8128-03c52c0dccef","Type":"ContainerStarted","Data":"060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c"} Sep 16 18:50:51.480032 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:51.479642 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547d7b6b54-vgl9d" event={"ID":"adfd869f-6302-4b33-8128-03c52c0dccef","Type":"ContainerStarted","Data":"9800d3d8fddb26a00c4187261c54c70e188b27266557ea4b7f8234663709ad6a"} Sep 16 18:50:51.501283 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:50:51.501220 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-547d7b6b54-vgl9d" podStartSLOduration=1.501198244 podStartE2EDuration="1.501198244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:50:51.499183015 +0000 UTC m=+339.042488775" watchObservedRunningTime="2026-09-16 18:50:51.501198244 +0000 UTC m=+339.044504003" Sep 16 18:51:00.742846 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:00.742804 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:51:00.743374 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:00.742890 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:51:00.747681 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:00.747657 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:51:01.525167 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:01.525131 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:51:01.585976 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:01.585943 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554c748f5d-6n2mn"] Sep 16 18:51:26.606074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.606023 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-554c748f5d-6n2mn" podUID="8f73caf1-9560-4516-a5fc-b805717168f8" containerName="console" containerID="cri-o://cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3" gracePeriod=15 Sep 16 18:51:26.844146 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.844120 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:51:26.907356 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907272 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9w52\" (UniqueName: \"kubernetes.io/projected/8f73caf1-9560-4516-a5fc-b805717168f8-kube-api-access-g9w52\") pod \"8f73caf1-9560-4516-a5fc-b805717168f8\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " Sep 16 18:51:26.907356 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907325 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-oauth-serving-cert\") pod \"8f73caf1-9560-4516-a5fc-b805717168f8\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " Sep 16 18:51:26.907356 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907343 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-serving-cert\") pod \"8f73caf1-9560-4516-a5fc-b805717168f8\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " Sep 16 18:51:26.907588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907375 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-console-config\") pod \"8f73caf1-9560-4516-a5fc-b805717168f8\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " Sep 16 18:51:26.907588 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907547 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-service-ca\") pod \"8f73caf1-9560-4516-a5fc-b805717168f8\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " Sep 16 18:51:26.907687 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907597 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-oauth-config\") pod \"8f73caf1-9560-4516-a5fc-b805717168f8\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " Sep 16 18:51:26.907687 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907647 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-trusted-ca-bundle\") pod \"8f73caf1-9560-4516-a5fc-b805717168f8\" (UID: \"8f73caf1-9560-4516-a5fc-b805717168f8\") " Sep 16 18:51:26.907796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907716 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-console-config" (OuterVolumeSpecName: "console-config") pod "8f73caf1-9560-4516-a5fc-b805717168f8" (UID: "8f73caf1-9560-4516-a5fc-b805717168f8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:51:26.907796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907734 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8f73caf1-9560-4516-a5fc-b805717168f8" (UID: "8f73caf1-9560-4516-a5fc-b805717168f8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:51:26.907967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.907939 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-service-ca" (OuterVolumeSpecName: "service-ca") pod "8f73caf1-9560-4516-a5fc-b805717168f8" (UID: "8f73caf1-9560-4516-a5fc-b805717168f8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:51:26.908032 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.908001 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8f73caf1-9560-4516-a5fc-b805717168f8" (UID: "8f73caf1-9560-4516-a5fc-b805717168f8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:51:26.908083 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.908063 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-oauth-serving-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:51:26.908083 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.908074 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-console-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:51:26.908083 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.908083 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-service-ca\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:51:26.908175 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.908091 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f73caf1-9560-4516-a5fc-b805717168f8-trusted-ca-bundle\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:51:26.909670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.909645 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f73caf1-9560-4516-a5fc-b805717168f8-kube-api-access-g9w52" (OuterVolumeSpecName: "kube-api-access-g9w52") pod "8f73caf1-9560-4516-a5fc-b805717168f8" (UID: "8f73caf1-9560-4516-a5fc-b805717168f8"). InnerVolumeSpecName "kube-api-access-g9w52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:51:26.909670 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.909644 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8f73caf1-9560-4516-a5fc-b805717168f8" (UID: "8f73caf1-9560-4516-a5fc-b805717168f8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:51:26.909942 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:26.909920 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8f73caf1-9560-4516-a5fc-b805717168f8" (UID: "8f73caf1-9560-4516-a5fc-b805717168f8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:51:27.008520 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.008481 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9w52\" (UniqueName: \"kubernetes.io/projected/8f73caf1-9560-4516-a5fc-b805717168f8-kube-api-access-g9w52\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:51:27.008520 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.008515 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-serving-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:51:27.008520 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.008524 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8f73caf1-9560-4516-a5fc-b805717168f8-console-oauth-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:51:27.626816 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.626781 2570 generic.go:356] "Generic (PLEG): container finished" podID="8f73caf1-9560-4516-a5fc-b805717168f8" containerID="cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3" exitCode=0 Sep 16 18:51:27.627275 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.626828 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554c748f5d-6n2mn" Sep 16 18:51:27.627275 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.626840 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554c748f5d-6n2mn" event={"ID":"8f73caf1-9560-4516-a5fc-b805717168f8","Type":"ContainerDied","Data":"cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3"} Sep 16 18:51:27.627275 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.626906 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554c748f5d-6n2mn" event={"ID":"8f73caf1-9560-4516-a5fc-b805717168f8","Type":"ContainerDied","Data":"9bbf5dbff888bb557a84a4814cf1b908669985295f5774959837bee7fcfe849f"} Sep 16 18:51:27.627275 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.626927 2570 scope.go:117] "RemoveContainer" containerID="cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3" Sep 16 18:51:27.635344 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.635326 2570 scope.go:117] "RemoveContainer" containerID="cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3" Sep 16 18:51:27.635702 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:51:27.635681 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3\": container with ID starting with cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3 not found: ID does not exist" containerID="cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3" Sep 16 18:51:27.635754 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.635711 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3"} err="failed to get container status \"cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3\": rpc error: code = NotFound desc = could not find container \"cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3\": container with ID starting with cd429b254a603385f6c92f2eefa2952f1455a72e0456bb8161232e03abb035c3 not found: ID does not exist" Sep 16 18:51:27.644279 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.644257 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-554c748f5d-6n2mn"] Sep 16 18:51:27.650341 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:27.650318 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-554c748f5d-6n2mn"] Sep 16 18:51:28.971091 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:51:28.971055 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f73caf1-9560-4516-a5fc-b805717168f8" path="/var/lib/kubelet/pods/8f73caf1-9560-4516-a5fc-b805717168f8/volumes" Sep 16 18:52:01.467737 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.467693 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk"] Sep 16 18:52:01.468328 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.468189 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f73caf1-9560-4516-a5fc-b805717168f8" containerName="console" Sep 16 18:52:01.468328 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.468207 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f73caf1-9560-4516-a5fc-b805717168f8" containerName="console" Sep 16 18:52:01.468328 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.468286 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f73caf1-9560-4516-a5fc-b805717168f8" containerName="console" Sep 16 18:52:01.471179 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.471158 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.477726 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.477394 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 16 18:52:01.478087 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.478064 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 16 18:52:01.479046 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.479027 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7sh2k\"" Sep 16 18:52:01.493738 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.493712 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk"] Sep 16 18:52:01.602404 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.602357 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.602404 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.602403 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.602618 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.602435 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsblw\" (UniqueName: \"kubernetes.io/projected/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-kube-api-access-tsblw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.703135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.703092 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.703135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.703134 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.703360 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.703152 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsblw\" (UniqueName: \"kubernetes.io/projected/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-kube-api-access-tsblw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.703527 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.703505 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.703527 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.703520 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.716827 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.716803 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsblw\" (UniqueName: \"kubernetes.io/projected/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-kube-api-access-tsblw\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:01.780069 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:01.779979 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:02.068565 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:02.068485 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk"] Sep 16 18:52:02.072445 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:52:02.072419 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9d241b_2d41_47d4_8791_1ff6ef9e0326.slice/crio-091b1c5afa99a0f9685672b34d912dbc3296b9dc9c44ee22026042920bdd6f51 WatchSource:0}: Error finding container 091b1c5afa99a0f9685672b34d912dbc3296b9dc9c44ee22026042920bdd6f51: Status 404 returned error can't find the container with id 091b1c5afa99a0f9685672b34d912dbc3296b9dc9c44ee22026042920bdd6f51 Sep 16 18:52:02.074493 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:02.074476 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:52:02.780637 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:02.780593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" event={"ID":"3c9d241b-2d41-47d4-8791-1ff6ef9e0326","Type":"ContainerStarted","Data":"091b1c5afa99a0f9685672b34d912dbc3296b9dc9c44ee22026042920bdd6f51"} Sep 16 18:52:09.814495 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:09.814449 2570 generic.go:356] "Generic (PLEG): container finished" podID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerID="f6cb4686422c79ace4418475eb0bde3a47734932f9646aef3bb59ff06684fcc4" exitCode=0 Sep 16 18:52:09.814982 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:09.814535 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" event={"ID":"3c9d241b-2d41-47d4-8791-1ff6ef9e0326","Type":"ContainerDied","Data":"f6cb4686422c79ace4418475eb0bde3a47734932f9646aef3bb59ff06684fcc4"} Sep 16 18:52:12.828967 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:12.828932 2570 generic.go:356] "Generic (PLEG): container finished" podID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerID="5df2f525e28104e229005bfc4db3cbcd05a2ee512c93d5a6b72dd6ef175ce589" exitCode=0 Sep 16 18:52:12.829362 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:12.829016 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" event={"ID":"3c9d241b-2d41-47d4-8791-1ff6ef9e0326","Type":"ContainerDied","Data":"5df2f525e28104e229005bfc4db3cbcd05a2ee512c93d5a6b72dd6ef175ce589"} Sep 16 18:52:19.866459 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:19.864728 2570 generic.go:356] "Generic (PLEG): container finished" podID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerID="c9df2de69e0d7a77f57b568c84b5ce776d3c9a72057c6ee94d9e2c18e4fa4097" exitCode=0 Sep 16 18:52:19.866459 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:19.864823 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" event={"ID":"3c9d241b-2d41-47d4-8791-1ff6ef9e0326","Type":"ContainerDied","Data":"c9df2de69e0d7a77f57b568c84b5ce776d3c9a72057c6ee94d9e2c18e4fa4097"} Sep 16 18:52:20.995258 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:20.995232 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:21.070337 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.070302 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-util\") pod \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " Sep 16 18:52:21.070511 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.070362 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-bundle\") pod \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " Sep 16 18:52:21.071019 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.070994 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-bundle" (OuterVolumeSpecName: "bundle") pod "3c9d241b-2d41-47d4-8791-1ff6ef9e0326" (UID: "3c9d241b-2d41-47d4-8791-1ff6ef9e0326"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:52:21.074537 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.074496 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-util" (OuterVolumeSpecName: "util") pod "3c9d241b-2d41-47d4-8791-1ff6ef9e0326" (UID: "3c9d241b-2d41-47d4-8791-1ff6ef9e0326"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:52:21.170935 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.170823 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsblw\" (UniqueName: \"kubernetes.io/projected/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-kube-api-access-tsblw\") pod \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\" (UID: \"3c9d241b-2d41-47d4-8791-1ff6ef9e0326\") " Sep 16 18:52:21.171085 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.171023 2570 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-bundle\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:52:21.171085 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.171034 2570 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-util\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:52:21.173201 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.173169 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-kube-api-access-tsblw" (OuterVolumeSpecName: "kube-api-access-tsblw") pod "3c9d241b-2d41-47d4-8791-1ff6ef9e0326" (UID: "3c9d241b-2d41-47d4-8791-1ff6ef9e0326"). InnerVolumeSpecName "kube-api-access-tsblw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:52:21.272331 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.272292 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsblw\" (UniqueName: \"kubernetes.io/projected/3c9d241b-2d41-47d4-8791-1ff6ef9e0326-kube-api-access-tsblw\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:52:21.875968 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.875941 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" Sep 16 18:52:21.875968 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.875957 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1adxsmk" event={"ID":"3c9d241b-2d41-47d4-8791-1ff6ef9e0326","Type":"ContainerDied","Data":"091b1c5afa99a0f9685672b34d912dbc3296b9dc9c44ee22026042920bdd6f51"} Sep 16 18:52:21.876170 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:21.875997 2570 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="091b1c5afa99a0f9685672b34d912dbc3296b9dc9c44ee22026042920bdd6f51" Sep 16 18:52:30.494446 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.494407 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b"] Sep 16 18:52:30.495066 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.494884 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerName="pull" Sep 16 18:52:30.495066 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.494903 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerName="pull" Sep 16 18:52:30.495066 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.494925 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerName="extract" Sep 16 18:52:30.495066 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.494933 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerName="extract" Sep 16 18:52:30.495066 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.494957 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerName="util" Sep 16 18:52:30.495066 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.494966 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerName="util" Sep 16 18:52:30.495066 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.495045 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c9d241b-2d41-47d4-8791-1ff6ef9e0326" containerName="extract" Sep 16 18:52:30.499347 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.499323 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:30.506650 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.506631 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 16 18:52:30.506753 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.506687 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 16 18:52:30.506753 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.506722 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-blzzr\"" Sep 16 18:52:30.508992 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.508974 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 16 18:52:30.537467 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.537432 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b"] Sep 16 18:52:30.552716 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.552678 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d89e91cf-03c0-457b-9d38-bfaf88433522-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4g46b\" (UID: \"d89e91cf-03c0-457b-9d38-bfaf88433522\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:30.552899 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.552748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphpd\" (UniqueName: \"kubernetes.io/projected/d89e91cf-03c0-457b-9d38-bfaf88433522-kube-api-access-wphpd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4g46b\" (UID: \"d89e91cf-03c0-457b-9d38-bfaf88433522\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:30.653843 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.653803 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphpd\" (UniqueName: \"kubernetes.io/projected/d89e91cf-03c0-457b-9d38-bfaf88433522-kube-api-access-wphpd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4g46b\" (UID: \"d89e91cf-03c0-457b-9d38-bfaf88433522\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:30.654115 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.653901 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d89e91cf-03c0-457b-9d38-bfaf88433522-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4g46b\" (UID: \"d89e91cf-03c0-457b-9d38-bfaf88433522\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:30.656425 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.656403 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/d89e91cf-03c0-457b-9d38-bfaf88433522-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4g46b\" (UID: \"d89e91cf-03c0-457b-9d38-bfaf88433522\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:30.664196 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.664167 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphpd\" (UniqueName: \"kubernetes.io/projected/d89e91cf-03c0-457b-9d38-bfaf88433522-kube-api-access-wphpd\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-4g46b\" (UID: \"d89e91cf-03c0-457b-9d38-bfaf88433522\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:30.809280 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:30.809189 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:31.105940 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:31.105914 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b"] Sep 16 18:52:31.108700 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:52:31.108671 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89e91cf_03c0_457b_9d38_bfaf88433522.slice/crio-43a63627fa3955a73c649e8cfcb19e1f3798de4649b975aebd54d297db830dae WatchSource:0}: Error finding container 43a63627fa3955a73c649e8cfcb19e1f3798de4649b975aebd54d297db830dae: Status 404 returned error can't find the container with id 43a63627fa3955a73c649e8cfcb19e1f3798de4649b975aebd54d297db830dae Sep 16 18:52:31.927583 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:31.927542 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" event={"ID":"d89e91cf-03c0-457b-9d38-bfaf88433522","Type":"ContainerStarted","Data":"43a63627fa3955a73c649e8cfcb19e1f3798de4649b975aebd54d297db830dae"} Sep 16 18:52:34.835713 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.835675 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-88kdd"] Sep 16 18:52:34.838041 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.838014 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:34.840506 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.840478 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Sep 16 18:52:34.840635 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.840524 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 16 18:52:34.840742 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.840727 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-z6rp6\"" Sep 16 18:52:34.851170 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.851138 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-88kdd"] Sep 16 18:52:34.894805 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.894764 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:34.894805 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.894808 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn5m\" (UniqueName: \"kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-kube-api-access-lpn5m\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:34.895082 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.894944 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/89e0df4a-5a1d-4a9d-869c-e8a83c906792-cabundle0\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:34.944625 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.944593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" event={"ID":"d89e91cf-03c0-457b-9d38-bfaf88433522","Type":"ContainerStarted","Data":"8f05c3db8a2d3ccbe57bc5fa1929717c560c802b7141eb5e1eb1e2adf17bd9d7"} Sep 16 18:52:34.944791 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.944731 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:52:34.967480 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.967429 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" podStartSLOduration=2.141326596 podStartE2EDuration="4.967413257s" totalImagesPullingTime="2.826086661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:34.964562432 +0000 UTC m=+442.507868185" watchObservedRunningTime="2026-09-16 18:52:34.967413257 +0000 UTC m=+442.510719015" Sep 16 18:52:34.995827 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.995792 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/89e0df4a-5a1d-4a9d-869c-e8a83c906792-cabundle0\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:34.996014 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.995891 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:34.996014 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.995935 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn5m\" (UniqueName: \"kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-kube-api-access-lpn5m\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:34.996014 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:34.995989 2570 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found Sep 16 18:52:34.996014 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:34.996007 2570 secret.go:281] references non-existent secret key: ca.crt Sep 16 18:52:34.996175 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:34.996018 2570 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 16 18:52:34.996175 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:34.996034 2570 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-88kdd: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 16 18:52:34.996175 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:34.996093 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates podName:89e0df4a-5a1d-4a9d-869c-e8a83c906792 nodeName:}" failed. No retries permitted until 2026-09-16 18:52:35.496075628 +0000 UTC m=+443.039381368 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates") pod "keda-operator-579c8958f7-88kdd" (UID: "89e0df4a-5a1d-4a9d-869c-e8a83c906792") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt] Sep 16 18:52:34.996513 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:34.996495 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/89e0df4a-5a1d-4a9d-869c-e8a83c906792-cabundle0\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:35.005401 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.005370 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn5m\" (UniqueName: \"kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-kube-api-access-lpn5m\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:35.115510 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.115428 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s"] Sep 16 18:52:35.117705 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.117689 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.126926 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.126900 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 16 18:52:35.142038 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.142002 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s"] Sep 16 18:52:35.198296 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.198257 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.198535 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.198307 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvhf\" (UniqueName: \"kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-kube-api-access-hpvhf\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.198535 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.198446 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/474a749a-ff87-410c-807c-488cb03ff311-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.299796 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.299752 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.300025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.299805 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvhf\" (UniqueName: \"kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-kube-api-access-hpvhf\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.300025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.299895 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/474a749a-ff87-410c-807c-488cb03ff311-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.300025 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.299938 2570 secret.go:281] references non-existent secret key: tls.crt Sep 16 18:52:35.300025 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.299965 2570 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Sep 16 18:52:35.300025 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.299989 2570 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s: references non-existent secret key: tls.crt Sep 16 18:52:35.300290 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.300068 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates podName:474a749a-ff87-410c-807c-488cb03ff311 nodeName:}" failed. No retries permitted until 2026-09-16 18:52:35.800047757 +0000 UTC m=+443.343353516 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates") pod "keda-metrics-apiserver-77857b7b98-w7j2s" (UID: "474a749a-ff87-410c-807c-488cb03ff311") : references non-existent secret key: tls.crt Sep 16 18:52:35.300290 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.300220 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/474a749a-ff87-410c-807c-488cb03ff311-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.310674 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.310639 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvhf\" (UniqueName: \"kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-kube-api-access-hpvhf\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.501571 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.501531 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:35.502081 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.501698 2570 secret.go:281] references non-existent secret key: ca.crt Sep 16 18:52:35.502081 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.501721 2570 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Sep 16 18:52:35.502081 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.501732 2570 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-88kdd: references non-existent secret key: ca.crt Sep 16 18:52:35.502081 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.501801 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates podName:89e0df4a-5a1d-4a9d-869c-e8a83c906792 nodeName:}" failed. No retries permitted until 2026-09-16 18:52:36.501781453 +0000 UTC m=+444.045087210 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates") pod "keda-operator-579c8958f7-88kdd" (UID: "89e0df4a-5a1d-4a9d-869c-e8a83c906792") : references non-existent secret key: ca.crt Sep 16 18:52:35.803836 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:35.803739 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:35.804364 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.804338 2570 secret.go:281] references non-existent secret key: tls.crt Sep 16 18:52:35.804515 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.804503 2570 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt Sep 16 18:52:35.804609 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.804600 2570 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s: references non-existent secret key: tls.crt Sep 16 18:52:35.804753 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:52:35.804743 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates podName:474a749a-ff87-410c-807c-488cb03ff311 nodeName:}" failed. No retries permitted until 2026-09-16 18:52:36.804721308 +0000 UTC m=+444.348027049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates") pod "keda-metrics-apiserver-77857b7b98-w7j2s" (UID: "474a749a-ff87-410c-807c-488cb03ff311") : references non-existent secret key: tls.crt Sep 16 18:52:36.510074 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:36.510030 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:36.512770 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:36.512735 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/89e0df4a-5a1d-4a9d-869c-e8a83c906792-certificates\") pod \"keda-operator-579c8958f7-88kdd\" (UID: \"89e0df4a-5a1d-4a9d-869c-e8a83c906792\") " pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:36.648747 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:36.648706 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:36.812975 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:36.812844 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:36.815645 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:36.815617 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/474a749a-ff87-410c-807c-488cb03ff311-certificates\") pod \"keda-metrics-apiserver-77857b7b98-w7j2s\" (UID: \"474a749a-ff87-410c-807c-488cb03ff311\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:36.926892 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:36.926771 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-88kdd"] Sep 16 18:52:36.927647 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:36.927628 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:36.929598 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:52:36.929570 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e0df4a_5a1d_4a9d_869c_e8a83c906792.slice/crio-45e86a7c325b3e900f0249bd65e057a59f418d91cd78071e847f20a0095d6012 WatchSource:0}: Error finding container 45e86a7c325b3e900f0249bd65e057a59f418d91cd78071e847f20a0095d6012: Status 404 returned error can't find the container with id 45e86a7c325b3e900f0249bd65e057a59f418d91cd78071e847f20a0095d6012 Sep 16 18:52:36.955581 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:36.955548 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-88kdd" event={"ID":"89e0df4a-5a1d-4a9d-869c-e8a83c906792","Type":"ContainerStarted","Data":"45e86a7c325b3e900f0249bd65e057a59f418d91cd78071e847f20a0095d6012"} Sep 16 18:52:37.211560 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:37.211457 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s"] Sep 16 18:52:37.215202 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:52:37.215173 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474a749a_ff87_410c_807c_488cb03ff311.slice/crio-af50b586d5e275793b17522d233648d622e4798f11215ad0b105269f69d747a8 WatchSource:0}: Error finding container af50b586d5e275793b17522d233648d622e4798f11215ad0b105269f69d747a8: Status 404 returned error can't find the container with id af50b586d5e275793b17522d233648d622e4798f11215ad0b105269f69d747a8 Sep 16 18:52:37.963701 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:37.963623 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" event={"ID":"474a749a-ff87-410c-807c-488cb03ff311","Type":"ContainerStarted","Data":"af50b586d5e275793b17522d233648d622e4798f11215ad0b105269f69d747a8"} Sep 16 18:52:40.980059 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:40.980019 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-88kdd" event={"ID":"89e0df4a-5a1d-4a9d-869c-e8a83c906792","Type":"ContainerStarted","Data":"04e621a4afbc39af4e259185b717803f1343015be6048c97f32a3565ac3f33b8"} Sep 16 18:52:40.980518 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:40.980089 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:52:40.981541 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:40.981518 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" event={"ID":"474a749a-ff87-410c-807c-488cb03ff311","Type":"ContainerStarted","Data":"259ec83dc2b6d9ebbcb294c9f3593068789463ca108c6f26fdea1178394b138b"} Sep 16 18:52:40.981671 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:40.981654 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:41.002636 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:41.002580 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-88kdd" podStartSLOduration=3.159530965 podStartE2EDuration="7.002561017s" totalImagesPullingTime="3.843030052s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:41.001553778 +0000 UTC m=+448.544859534" watchObservedRunningTime="2026-09-16 18:52:41.002561017 +0000 UTC m=+448.545866775" Sep 16 18:52:41.029450 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:41.029342 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" podStartSLOduration=2.476011535 podStartE2EDuration="6.029323081s" totalImagesPullingTime="3.553311546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:41.027819237 +0000 UTC m=+448.571125001" watchObservedRunningTime="2026-09-16 18:52:41.029323081 +0000 UTC m=+448.572628840" Sep 16 18:52:51.992645 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:51.992618 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-w7j2s" Sep 16 18:52:55.951593 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:52:55.951559 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-4g46b" Sep 16 18:53:01.991017 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:53:01.990986 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-88kdd" Sep 16 18:54:00.257024 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.256982 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-t2ztr"] Sep 16 18:54:00.261124 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.261092 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:00.265146 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.265121 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 18:54:00.265146 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.265130 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-4phrp\"" Sep 16 18:54:00.265482 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.265465 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 16 18:54:00.265568 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.265553 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 18:54:00.271640 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.271613 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-t2ztr"] Sep 16 18:54:00.278145 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.278118 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr"] Sep 16 18:54:00.280619 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.280597 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:00.283170 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.283150 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-xh998\"" Sep 16 18:54:00.283332 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.283316 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Sep 16 18:54:00.298153 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.298118 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr"] Sep 16 18:54:00.331500 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.331465 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrpv\" (UniqueName: \"kubernetes.io/projected/6cfa057a-4f53-4639-bbc2-a81daeebc6b9-kube-api-access-5zrpv\") pod \"llmisvc-controller-manager-645f9c8fd6-fz4mr\" (UID: \"6cfa057a-4f53-4639-bbc2-a81daeebc6b9\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:00.331500 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.331506 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhlb\" (UniqueName: \"kubernetes.io/projected/7ac536e8-56ef-4da5-be6e-0f62f235b079-kube-api-access-vwhlb\") pod \"kserve-controller-manager-764fc98f48-t2ztr\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:00.331775 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.331529 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert\") pod \"kserve-controller-manager-764fc98f48-t2ztr\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:00.331775 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.331591 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cfa057a-4f53-4639-bbc2-a81daeebc6b9-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-fz4mr\" (UID: \"6cfa057a-4f53-4639-bbc2-a81daeebc6b9\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:00.432138 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.432094 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert\") pod \"kserve-controller-manager-764fc98f48-t2ztr\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:00.432138 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.432144 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cfa057a-4f53-4639-bbc2-a81daeebc6b9-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-fz4mr\" (UID: \"6cfa057a-4f53-4639-bbc2-a81daeebc6b9\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:00.432353 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.432212 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrpv\" (UniqueName: \"kubernetes.io/projected/6cfa057a-4f53-4639-bbc2-a81daeebc6b9-kube-api-access-5zrpv\") pod \"llmisvc-controller-manager-645f9c8fd6-fz4mr\" (UID: \"6cfa057a-4f53-4639-bbc2-a81daeebc6b9\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:00.432353 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.432242 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhlb\" (UniqueName: \"kubernetes.io/projected/7ac536e8-56ef-4da5-be6e-0f62f235b079-kube-api-access-vwhlb\") pod \"kserve-controller-manager-764fc98f48-t2ztr\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:00.432353 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:54:00.432256 2570 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Sep 16 18:54:00.432353 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:54:00.432344 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert podName:7ac536e8-56ef-4da5-be6e-0f62f235b079 nodeName:}" failed. No retries permitted until 2026-09-16 18:54:00.932328244 +0000 UTC m=+528.475633981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert") pod "kserve-controller-manager-764fc98f48-t2ztr" (UID: "7ac536e8-56ef-4da5-be6e-0f62f235b079") : secret "kserve-webhook-server-cert" not found Sep 16 18:54:00.435017 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.434978 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6cfa057a-4f53-4639-bbc2-a81daeebc6b9-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-fz4mr\" (UID: \"6cfa057a-4f53-4639-bbc2-a81daeebc6b9\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:00.441273 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.441242 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhlb\" (UniqueName: \"kubernetes.io/projected/7ac536e8-56ef-4da5-be6e-0f62f235b079-kube-api-access-vwhlb\") pod \"kserve-controller-manager-764fc98f48-t2ztr\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:00.441624 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.441601 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrpv\" (UniqueName: \"kubernetes.io/projected/6cfa057a-4f53-4639-bbc2-a81daeebc6b9-kube-api-access-5zrpv\") pod \"llmisvc-controller-manager-645f9c8fd6-fz4mr\" (UID: \"6cfa057a-4f53-4639-bbc2-a81daeebc6b9\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:00.591655 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.591559 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:00.866984 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.866960 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr"] Sep 16 18:54:00.869517 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:54:00.869485 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6cfa057a_4f53_4639_bbc2_a81daeebc6b9.slice/crio-f126ffcdd6f49b0b1b8fc94a9c94264298ffad084273a454c03dcef5c4c90fde WatchSource:0}: Error finding container f126ffcdd6f49b0b1b8fc94a9c94264298ffad084273a454c03dcef5c4c90fde: Status 404 returned error can't find the container with id f126ffcdd6f49b0b1b8fc94a9c94264298ffad084273a454c03dcef5c4c90fde Sep 16 18:54:00.935124 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.935086 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert\") pod \"kserve-controller-manager-764fc98f48-t2ztr\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:00.937744 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:00.937715 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert\") pod \"kserve-controller-manager-764fc98f48-t2ztr\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:01.173153 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:01.173066 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:01.358932 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:01.358893 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" event={"ID":"6cfa057a-4f53-4639-bbc2-a81daeebc6b9","Type":"ContainerStarted","Data":"f126ffcdd6f49b0b1b8fc94a9c94264298ffad084273a454c03dcef5c4c90fde"} Sep 16 18:54:01.448756 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:01.448714 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-t2ztr"] Sep 16 18:54:01.451983 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:54:01.451953 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac536e8_56ef_4da5_be6e_0f62f235b079.slice/crio-c43a46448e57dc2361ba552b0d76f1f2b4cdd6d46b07c36d11b7c18516fda705 WatchSource:0}: Error finding container c43a46448e57dc2361ba552b0d76f1f2b4cdd6d46b07c36d11b7c18516fda705: Status 404 returned error can't find the container with id c43a46448e57dc2361ba552b0d76f1f2b4cdd6d46b07c36d11b7c18516fda705 Sep 16 18:54:02.368249 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:02.368191 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" event={"ID":"7ac536e8-56ef-4da5-be6e-0f62f235b079","Type":"ContainerStarted","Data":"c43a46448e57dc2361ba552b0d76f1f2b4cdd6d46b07c36d11b7c18516fda705"} Sep 16 18:54:04.380948 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:04.380908 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" event={"ID":"7ac536e8-56ef-4da5-be6e-0f62f235b079","Type":"ContainerStarted","Data":"9f314abc365994b93d3ad36154da1ac2d1b2e0c556c5f0c979ed215f93932970"} Sep 16 18:54:04.381408 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:04.381020 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:04.382662 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:04.382638 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" event={"ID":"6cfa057a-4f53-4639-bbc2-a81daeebc6b9","Type":"ContainerStarted","Data":"b730f1d2c67c38e88a2a243fc22d8688bba32be7bbbf84b5d97fbceb922a59fe"} Sep 16 18:54:04.382805 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:04.382755 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:04.400940 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:04.400888 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" podStartSLOduration=1.665414992 podStartE2EDuration="4.400847779s" totalImagesPullingTime="2.735432787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:04.399283033 +0000 UTC m=+531.942588798" watchObservedRunningTime="2026-09-16 18:54:04.400847779 +0000 UTC m=+531.944153537" Sep 16 18:54:04.417399 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:04.417350 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" podStartSLOduration=1.100312539 podStartE2EDuration="4.417332282s" totalImagesPullingTime="3.317019743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:04.416281761 +0000 UTC m=+531.959587512" watchObservedRunningTime="2026-09-16 18:54:04.417332282 +0000 UTC m=+531.960638083" Sep 16 18:54:35.390466 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:35.390432 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-fz4mr" Sep 16 18:54:35.393584 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:35.393564 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:44.363204 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.363166 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-t2ztr"] Sep 16 18:54:44.363601 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.363354 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" podUID="7ac536e8-56ef-4da5-be6e-0f62f235b079" containerName="manager" containerID="cri-o://9f314abc365994b93d3ad36154da1ac2d1b2e0c556c5f0c979ed215f93932970" gracePeriod=10 Sep 16 18:54:44.386660 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.386629 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wrrs4"] Sep 16 18:54:44.388931 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.388912 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:44.398523 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.398491 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wrrs4"] Sep 16 18:54:44.521675 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.521629 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96012204-f109-4e7f-932c-ab5c0f39ffcc-cert\") pod \"kserve-controller-manager-764fc98f48-wrrs4\" (UID: \"96012204-f109-4e7f-932c-ab5c0f39ffcc\") " pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:44.521878 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.521720 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjpzs\" (UniqueName: \"kubernetes.io/projected/96012204-f109-4e7f-932c-ab5c0f39ffcc-kube-api-access-jjpzs\") pod \"kserve-controller-manager-764fc98f48-wrrs4\" (UID: \"96012204-f109-4e7f-932c-ab5c0f39ffcc\") " pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:44.568239 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.568208 2570 generic.go:356] "Generic (PLEG): container finished" podID="7ac536e8-56ef-4da5-be6e-0f62f235b079" containerID="9f314abc365994b93d3ad36154da1ac2d1b2e0c556c5f0c979ed215f93932970" exitCode=0 Sep 16 18:54:44.568403 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.568278 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" event={"ID":"7ac536e8-56ef-4da5-be6e-0f62f235b079","Type":"ContainerDied","Data":"9f314abc365994b93d3ad36154da1ac2d1b2e0c556c5f0c979ed215f93932970"} Sep 16 18:54:44.613690 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.613619 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:44.622937 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.622899 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjpzs\" (UniqueName: \"kubernetes.io/projected/96012204-f109-4e7f-932c-ab5c0f39ffcc-kube-api-access-jjpzs\") pod \"kserve-controller-manager-764fc98f48-wrrs4\" (UID: \"96012204-f109-4e7f-932c-ab5c0f39ffcc\") " pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:44.623072 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.623008 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96012204-f109-4e7f-932c-ab5c0f39ffcc-cert\") pod \"kserve-controller-manager-764fc98f48-wrrs4\" (UID: \"96012204-f109-4e7f-932c-ab5c0f39ffcc\") " pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:44.625438 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.625415 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/96012204-f109-4e7f-932c-ab5c0f39ffcc-cert\") pod \"kserve-controller-manager-764fc98f48-wrrs4\" (UID: \"96012204-f109-4e7f-932c-ab5c0f39ffcc\") " pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:44.633050 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.633024 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjpzs\" (UniqueName: \"kubernetes.io/projected/96012204-f109-4e7f-932c-ab5c0f39ffcc-kube-api-access-jjpzs\") pod \"kserve-controller-manager-764fc98f48-wrrs4\" (UID: \"96012204-f109-4e7f-932c-ab5c0f39ffcc\") " pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:44.723374 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.723331 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert\") pod \"7ac536e8-56ef-4da5-be6e-0f62f235b079\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " Sep 16 18:54:44.723544 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.723457 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwhlb\" (UniqueName: \"kubernetes.io/projected/7ac536e8-56ef-4da5-be6e-0f62f235b079-kube-api-access-vwhlb\") pod \"7ac536e8-56ef-4da5-be6e-0f62f235b079\" (UID: \"7ac536e8-56ef-4da5-be6e-0f62f235b079\") " Sep 16 18:54:44.725697 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.725666 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac536e8-56ef-4da5-be6e-0f62f235b079-kube-api-access-vwhlb" (OuterVolumeSpecName: "kube-api-access-vwhlb") pod "7ac536e8-56ef-4da5-be6e-0f62f235b079" (UID: "7ac536e8-56ef-4da5-be6e-0f62f235b079"). InnerVolumeSpecName "kube-api-access-vwhlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:54:44.725823 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.725709 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert" (OuterVolumeSpecName: "cert") pod "7ac536e8-56ef-4da5-be6e-0f62f235b079" (UID: "7ac536e8-56ef-4da5-be6e-0f62f235b079"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:54:44.729871 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.729840 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:44.824705 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.824663 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhlb\" (UniqueName: \"kubernetes.io/projected/7ac536e8-56ef-4da5-be6e-0f62f235b079-kube-api-access-vwhlb\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:54:44.824705 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:44.824696 2570 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac536e8-56ef-4da5-be6e-0f62f235b079-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:54:45.000411 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.000380 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-wrrs4"] Sep 16 18:54:45.003577 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:54:45.003541 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96012204_f109_4e7f_932c_ab5c0f39ffcc.slice/crio-d44529fa992925767e71400463b614f529421e3179d872b531a5085879ed4864 WatchSource:0}: Error finding container d44529fa992925767e71400463b614f529421e3179d872b531a5085879ed4864: Status 404 returned error can't find the container with id d44529fa992925767e71400463b614f529421e3179d872b531a5085879ed4864 Sep 16 18:54:45.574494 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.574451 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" event={"ID":"7ac536e8-56ef-4da5-be6e-0f62f235b079","Type":"ContainerDied","Data":"c43a46448e57dc2361ba552b0d76f1f2b4cdd6d46b07c36d11b7c18516fda705"} Sep 16 18:54:45.574988 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.574462 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-764fc98f48-t2ztr" Sep 16 18:54:45.574988 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.574554 2570 scope.go:117] "RemoveContainer" containerID="9f314abc365994b93d3ad36154da1ac2d1b2e0c556c5f0c979ed215f93932970" Sep 16 18:54:45.576272 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.576206 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" event={"ID":"96012204-f109-4e7f-932c-ab5c0f39ffcc","Type":"ContainerStarted","Data":"cde290650c22e7cd6d6cd8380b1ae0d94f916a97368565664a33ffab55c9257e"} Sep 16 18:54:45.576272 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.576246 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" event={"ID":"96012204-f109-4e7f-932c-ab5c0f39ffcc","Type":"ContainerStarted","Data":"d44529fa992925767e71400463b614f529421e3179d872b531a5085879ed4864"} Sep 16 18:54:45.576446 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.576285 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:54:45.589965 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.589941 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-t2ztr"] Sep 16 18:54:45.593636 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.593585 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-764fc98f48-t2ztr"] Sep 16 18:54:45.608069 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:45.608024 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" podStartSLOduration=1.270225808 podStartE2EDuration="1.608005815s" totalImagesPullingTime="337.780007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:45.606066428 +0000 UTC m=+573.149372168" watchObservedRunningTime="2026-09-16 18:54:45.608005815 +0000 UTC m=+573.151311585" Sep 16 18:54:46.971959 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:54:46.971925 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac536e8-56ef-4da5-be6e-0f62f235b079" path="/var/lib/kubelet/pods/7ac536e8-56ef-4da5-be6e-0f62f235b079/volumes" Sep 16 18:55:12.900322 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:12.900290 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:55:12.903365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:12.903338 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 18:55:16.588151 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:16.588112 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-764fc98f48-wrrs4" Sep 16 18:55:18.000610 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.000575 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-h4rhl"] Sep 16 18:55:18.001102 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.000972 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ac536e8-56ef-4da5-be6e-0f62f235b079" containerName="manager" Sep 16 18:55:18.001102 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.000985 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ac536e8-56ef-4da5-be6e-0f62f235b079" containerName="manager" Sep 16 18:55:18.001102 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.001040 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ac536e8-56ef-4da5-be6e-0f62f235b079" containerName="manager" Sep 16 18:55:18.008951 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.008922 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:18.011275 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.011224 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-hrtjn\"" Sep 16 18:55:18.011447 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.011403 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 16 18:55:18.014194 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.014164 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-h4rhl"] Sep 16 18:55:18.118313 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.118274 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552-tls-certs\") pod \"model-serving-api-7b954ffc95-h4rhl\" (UID: \"2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552\") " pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:18.118504 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.118339 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjxd\" (UniqueName: \"kubernetes.io/projected/2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552-kube-api-access-hfjxd\") pod \"model-serving-api-7b954ffc95-h4rhl\" (UID: \"2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552\") " pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:18.218919 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.218848 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552-tls-certs\") pod \"model-serving-api-7b954ffc95-h4rhl\" (UID: \"2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552\") " pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:18.219106 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.218970 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjxd\" (UniqueName: \"kubernetes.io/projected/2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552-kube-api-access-hfjxd\") pod \"model-serving-api-7b954ffc95-h4rhl\" (UID: \"2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552\") " pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:18.221941 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.221916 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552-tls-certs\") pod \"model-serving-api-7b954ffc95-h4rhl\" (UID: \"2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552\") " pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:18.227809 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.227777 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjxd\" (UniqueName: \"kubernetes.io/projected/2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552-kube-api-access-hfjxd\") pod \"model-serving-api-7b954ffc95-h4rhl\" (UID: \"2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552\") " pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:18.321120 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.320965 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:18.602653 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.602628 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-h4rhl"] Sep 16 18:55:18.604960 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:55:18.604926 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bac9c4b_3ce7_44ea_ac1d_190d8f1ca552.slice/crio-8562f02e2f65ad8b779a699f94ce87728944696eadf1e3b2344d9a93f8be9264 WatchSource:0}: Error finding container 8562f02e2f65ad8b779a699f94ce87728944696eadf1e3b2344d9a93f8be9264: Status 404 returned error can't find the container with id 8562f02e2f65ad8b779a699f94ce87728944696eadf1e3b2344d9a93f8be9264 Sep 16 18:55:18.734494 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:18.734460 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-h4rhl" event={"ID":"2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552","Type":"ContainerStarted","Data":"8562f02e2f65ad8b779a699f94ce87728944696eadf1e3b2344d9a93f8be9264"} Sep 16 18:55:21.751208 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:21.751165 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-h4rhl" event={"ID":"2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552","Type":"ContainerStarted","Data":"492130ac129373848fe4e5e41192daffea80d972ad0b7bb618515ba34222ecfc"} Sep 16 18:55:21.751690 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:21.751260 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:55:21.771270 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:21.771217 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-h4rhl" podStartSLOduration=2.613027871 podStartE2EDuration="4.771201149s" totalImagesPullingTime="2.158173278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:21.76938763 +0000 UTC m=+609.312693402" watchObservedRunningTime="2026-09-16 18:55:21.771201149 +0000 UTC m=+609.314506907" Sep 16 18:55:32.760734 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:55:32.760704 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-h4rhl" Sep 16 18:57:05.074397 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.074314 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp"] Sep 16 18:57:05.077886 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.077843 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.080065 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.080046 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dc809-predictor-serving-cert\"" Sep 16 18:57:05.080536 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.080519 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fxpzh\"" Sep 16 18:57:05.080835 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.080819 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:57:05.081083 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.081068 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:57:05.081459 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.081443 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-dc809-kube-rbac-proxy-sar-config\"" Sep 16 18:57:05.092016 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.091991 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp"] Sep 16 18:57:05.145788 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.145758 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbceae1-9896-4df9-af4c-05ecff945df6-proxy-tls\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.145976 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.145804 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hczp\" (UniqueName: \"kubernetes.io/projected/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-api-access-5hczp\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.145976 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.145884 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.246807 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.246774 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbceae1-9896-4df9-af4c-05ecff945df6-proxy-tls\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.247034 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.246834 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hczp\" (UniqueName: \"kubernetes.io/projected/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-api-access-5hczp\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.247034 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.246897 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.247608 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.247583 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.249515 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.249489 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbceae1-9896-4df9-af4c-05ecff945df6-proxy-tls\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.256541 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.256511 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hczp\" (UniqueName: \"kubernetes.io/projected/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-api-access-5hczp\") pod \"error-404-isvc-dc809-predictor-59c6f6455b-2r8hp\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.388894 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.388773 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:05.668799 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.668704 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp"] Sep 16 18:57:05.672172 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:57:05.672141 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfbceae1_9896_4df9_af4c_05ecff945df6.slice/crio-f9fc5a24976efb8a27675e50e3506e3eaf844203bdb0aac0f59169bde978705e WatchSource:0}: Error finding container f9fc5a24976efb8a27675e50e3506e3eaf844203bdb0aac0f59169bde978705e: Status 404 returned error can't find the container with id f9fc5a24976efb8a27675e50e3506e3eaf844203bdb0aac0f59169bde978705e Sep 16 18:57:05.674313 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:05.674295 2570 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:57:06.246277 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:06.246234 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" event={"ID":"bfbceae1-9896-4df9-af4c-05ecff945df6","Type":"ContainerStarted","Data":"f9fc5a24976efb8a27675e50e3506e3eaf844203bdb0aac0f59169bde978705e"} Sep 16 18:57:17.315055 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:17.315025 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" event={"ID":"bfbceae1-9896-4df9-af4c-05ecff945df6","Type":"ContainerStarted","Data":"4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7"} Sep 16 18:57:20.333441 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:20.333408 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" event={"ID":"bfbceae1-9896-4df9-af4c-05ecff945df6","Type":"ContainerStarted","Data":"b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700"} Sep 16 18:57:20.333915 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:20.333582 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:20.356881 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:20.356808 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podStartSLOduration=1.583118093 podStartE2EDuration="15.356789837s" totalImagesPullingTime="13.773671744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:20.356200184 +0000 UTC m=+727.899505948" watchObservedRunningTime="2026-09-16 18:57:20.356789837 +0000 UTC m=+727.900095676" Sep 16 18:57:21.338168 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:21.338138 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:21.339733 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:21.339705 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 18:57:22.342380 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:22.342338 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 18:57:27.346742 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:27.346706 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:57:27.347362 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:27.347332 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 18:57:37.347920 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:37.347852 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 18:57:47.347964 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:47.347919 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 18:57:57.348130 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:57:57.348092 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 18:58:07.349058 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:07.349027 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:58:35.008337 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.008254 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp"] Sep 16 18:58:35.009086 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.008520 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" containerID="cri-o://4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7" gracePeriod=30 Sep 16 18:58:35.009086 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.008561 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kube-rbac-proxy" containerID="cri-o://b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700" gracePeriod=30 Sep 16 18:58:35.201188 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.201143 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq"] Sep 16 18:58:35.204328 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.204304 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.206464 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.206441 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9a5c2-kube-rbac-proxy-sar-config\"" Sep 16 18:58:35.206591 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.206478 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9a5c2-predictor-serving-cert\"" Sep 16 18:58:35.213604 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.213582 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq"] Sep 16 18:58:35.253989 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.253942 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.253989 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.253990 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkdtn\" (UniqueName: \"kubernetes.io/projected/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-api-access-tkdtn\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.254226 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.254026 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.354523 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.354436 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.354523 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.354471 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkdtn\" (UniqueName: \"kubernetes.io/projected/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-api-access-tkdtn\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.354523 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.354493 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.354790 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:58:35.354605 2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-serving-cert: secret "error-404-isvc-9a5c2-predictor-serving-cert" not found Sep 16 18:58:35.354790 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:58:35.354678 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls podName:494bb4ca-014f-4713-aff2-6456f8b5a03e nodeName:}" failed. No retries permitted until 2026-09-16 18:58:35.854659204 +0000 UTC m=+803.397964943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls") pod "error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" (UID: "494bb4ca-014f-4713-aff2-6456f8b5a03e") : secret "error-404-isvc-9a5c2-predictor-serving-cert" not found Sep 16 18:58:35.355208 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.355190 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.363909 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.363848 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkdtn\" (UniqueName: \"kubernetes.io/projected/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-api-access-tkdtn\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.691259 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.691159 2570 generic.go:356] "Generic (PLEG): container finished" podID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerID="b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700" exitCode=2 Sep 16 18:58:35.691412 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.691265 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" event={"ID":"bfbceae1-9896-4df9-af4c-05ecff945df6","Type":"ContainerDied","Data":"b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700"} Sep 16 18:58:35.859334 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.859270 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:35.862077 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:35.862034 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls\") pod \"error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:36.115723 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:36.115687 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:36.394275 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:36.394250 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq"] Sep 16 18:58:36.396949 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:58:36.396920 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod494bb4ca_014f_4713_aff2_6456f8b5a03e.slice/crio-bd88ff2dee4afd56bc596c56d771b1081ff6a47b6c2799d33b3e3eec2dad5fc4 WatchSource:0}: Error finding container bd88ff2dee4afd56bc596c56d771b1081ff6a47b6c2799d33b3e3eec2dad5fc4: Status 404 returned error can't find the container with id bd88ff2dee4afd56bc596c56d771b1081ff6a47b6c2799d33b3e3eec2dad5fc4 Sep 16 18:58:36.698681 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:36.698569 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" event={"ID":"494bb4ca-014f-4713-aff2-6456f8b5a03e","Type":"ContainerStarted","Data":"98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd"} Sep 16 18:58:36.698681 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:36.698646 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" event={"ID":"494bb4ca-014f-4713-aff2-6456f8b5a03e","Type":"ContainerStarted","Data":"e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9"} Sep 16 18:58:36.698681 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:36.698677 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" event={"ID":"494bb4ca-014f-4713-aff2-6456f8b5a03e","Type":"ContainerStarted","Data":"bd88ff2dee4afd56bc596c56d771b1081ff6a47b6c2799d33b3e3eec2dad5fc4"} Sep 16 18:58:36.698935 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:36.698711 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:36.717816 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:36.717760 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podStartSLOduration=1.717740362 podStartE2EDuration="1.717740362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:36.716836204 +0000 UTC m=+804.260141964" watchObservedRunningTime="2026-09-16 18:58:36.717740362 +0000 UTC m=+804.261046120" Sep 16 18:58:37.343055 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:37.343010 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused" Sep 16 18:58:37.347254 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:37.347222 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused" Sep 16 18:58:37.703109 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:37.703033 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:37.704293 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:37.704270 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 18:58:38.450194 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.450167 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:58:38.582226 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.582140 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-rbac-proxy-sar-config\") pod \"bfbceae1-9896-4df9-af4c-05ecff945df6\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " Sep 16 18:58:38.582226 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.582186 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hczp\" (UniqueName: \"kubernetes.io/projected/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-api-access-5hczp\") pod \"bfbceae1-9896-4df9-af4c-05ecff945df6\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " Sep 16 18:58:38.582436 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.582239 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbceae1-9896-4df9-af4c-05ecff945df6-proxy-tls\") pod \"bfbceae1-9896-4df9-af4c-05ecff945df6\" (UID: \"bfbceae1-9896-4df9-af4c-05ecff945df6\") " Sep 16 18:58:38.582628 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.582596 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfbceae1-9896-4df9-af4c-05ecff945df6" (UID: "bfbceae1-9896-4df9-af4c-05ecff945df6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:58:38.584737 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.584701 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfbceae1-9896-4df9-af4c-05ecff945df6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfbceae1-9896-4df9-af4c-05ecff945df6" (UID: "bfbceae1-9896-4df9-af4c-05ecff945df6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:58:38.584842 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.584753 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-api-access-5hczp" (OuterVolumeSpecName: "kube-api-access-5hczp") pod "bfbceae1-9896-4df9-af4c-05ecff945df6" (UID: "bfbceae1-9896-4df9-af4c-05ecff945df6"). InnerVolumeSpecName "kube-api-access-5hczp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:58:38.683033 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.682997 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:58:38.683033 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.683029 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hczp\" (UniqueName: \"kubernetes.io/projected/bfbceae1-9896-4df9-af4c-05ecff945df6-kube-api-access-5hczp\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:58:38.683033 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.683042 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfbceae1-9896-4df9-af4c-05ecff945df6-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:58:38.715687 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.715656 2570 generic.go:356] "Generic (PLEG): container finished" podID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerID="4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7" exitCode=0 Sep 16 18:58:38.715856 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.715723 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" Sep 16 18:58:38.715856 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.715724 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" event={"ID":"bfbceae1-9896-4df9-af4c-05ecff945df6","Type":"ContainerDied","Data":"4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7"} Sep 16 18:58:38.715856 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.715827 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp" event={"ID":"bfbceae1-9896-4df9-af4c-05ecff945df6","Type":"ContainerDied","Data":"f9fc5a24976efb8a27675e50e3506e3eaf844203bdb0aac0f59169bde978705e"} Sep 16 18:58:38.716027 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.715884 2570 scope.go:117] "RemoveContainer" containerID="b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700" Sep 16 18:58:38.716325 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.716291 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 18:58:38.725257 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.725240 2570 scope.go:117] "RemoveContainer" containerID="4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7" Sep 16 18:58:38.735603 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.735581 2570 scope.go:117] "RemoveContainer" containerID="b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700" Sep 16 18:58:38.736054 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:58:38.736028 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700\": container with ID starting with b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700 not found: ID does not exist" containerID="b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700" Sep 16 18:58:38.736135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.736065 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700"} err="failed to get container status \"b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700\": rpc error: code = NotFound desc = could not find container \"b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700\": container with ID starting with b91487444140bebded05895a6ebf794623c597831041719c15b1380d9d212700 not found: ID does not exist" Sep 16 18:58:38.736135 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.736090 2570 scope.go:117] "RemoveContainer" containerID="4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7" Sep 16 18:58:38.736509 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:58:38.736491 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7\": container with ID starting with 4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7 not found: ID does not exist" containerID="4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7" Sep 16 18:58:38.736554 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.736525 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7"} err="failed to get container status \"4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7\": rpc error: code = NotFound desc = could not find container \"4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7\": container with ID starting with 4156400ca407c2e583fb106f4b39e22074330349ec64577be08ee423fbd86da7 not found: ID does not exist" Sep 16 18:58:38.737409 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.737388 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp"] Sep 16 18:58:38.741041 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.741015 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-dc809-predictor-59c6f6455b-2r8hp"] Sep 16 18:58:38.970906 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:38.970854 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" path="/var/lib/kubelet/pods/bfbceae1-9896-4df9-af4c-05ecff945df6/volumes" Sep 16 18:58:43.722388 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:43.722359 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:58:43.722805 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:43.722782 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 18:58:53.722878 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:58:53.722825 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 18:59:03.723524 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:03.723477 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 18:59:07.009087 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.009047 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6844df6d4c-ndbx2"] Sep 16 18:59:07.009578 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.009418 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" Sep 16 18:59:07.009578 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.009430 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" Sep 16 18:59:07.009578 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.009437 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kube-rbac-proxy" Sep 16 18:59:07.009578 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.009444 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kube-rbac-proxy" Sep 16 18:59:07.009578 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.009521 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kube-rbac-proxy" Sep 16 18:59:07.009578 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.009532 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfbceae1-9896-4df9-af4c-05ecff945df6" containerName="kserve-container" Sep 16 18:59:07.012065 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.012046 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.023099 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.023071 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6844df6d4c-ndbx2"] Sep 16 18:59:07.047455 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.047416 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-oauth-config\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.047455 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.047456 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l97n2\" (UniqueName: \"kubernetes.io/projected/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-kube-api-access-l97n2\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.047693 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.047486 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-oauth-serving-cert\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.047693 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.047527 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-trusted-ca-bundle\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.047693 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.047571 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-config\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.047693 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.047615 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-serving-cert\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.047693 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.047673 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-service-ca\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.148814 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.148777 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-serving-cert\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.148830 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-service-ca\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.148855 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-oauth-config\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.148894 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l97n2\" (UniqueName: \"kubernetes.io/projected/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-kube-api-access-l97n2\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.148927 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-oauth-serving-cert\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.148953 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-trusted-ca-bundle\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149025 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.148981 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-config\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149676 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.149644 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-service-ca\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149788 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.149693 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-config\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.149968 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.149944 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-oauth-serving-cert\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.150013 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.149995 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-trusted-ca-bundle\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.151497 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.151468 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-oauth-config\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.151589 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.151574 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-console-serving-cert\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.158970 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.158943 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l97n2\" (UniqueName: \"kubernetes.io/projected/c5b17e5a-7ae2-4279-af3a-3a28080b81c9-kube-api-access-l97n2\") pod \"console-6844df6d4c-ndbx2\" (UID: \"c5b17e5a-7ae2-4279-af3a-3a28080b81c9\") " pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.322807 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.322709 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:07.600077 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.599995 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6844df6d4c-ndbx2"] Sep 16 18:59:07.603082 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:59:07.603051 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b17e5a_7ae2_4279_af3a_3a28080b81c9.slice/crio-bc1fddb6cd2aff168ec5f7f4f7a4987fd48d99a0ffdb3537a0686a0b4aeb8216 WatchSource:0}: Error finding container bc1fddb6cd2aff168ec5f7f4f7a4987fd48d99a0ffdb3537a0686a0b4aeb8216: Status 404 returned error can't find the container with id bc1fddb6cd2aff168ec5f7f4f7a4987fd48d99a0ffdb3537a0686a0b4aeb8216 Sep 16 18:59:07.867607 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.867511 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6844df6d4c-ndbx2" event={"ID":"c5b17e5a-7ae2-4279-af3a-3a28080b81c9","Type":"ContainerStarted","Data":"d847bd31ab2d700d34843e39d41d34801675d1014ce21bd740c14095f8a303a5"} Sep 16 18:59:07.867607 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.867568 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6844df6d4c-ndbx2" event={"ID":"c5b17e5a-7ae2-4279-af3a-3a28080b81c9","Type":"ContainerStarted","Data":"bc1fddb6cd2aff168ec5f7f4f7a4987fd48d99a0ffdb3537a0686a0b4aeb8216"} Sep 16 18:59:07.889696 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:07.889633 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6844df6d4c-ndbx2" podStartSLOduration=1.8896162410000001 podStartE2EDuration="1.889616241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:07.887335138 +0000 UTC m=+835.430640910" watchObservedRunningTime="2026-09-16 18:59:07.889616241 +0000 UTC m=+835.432922000" Sep 16 18:59:13.723579 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:13.723534 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 16 18:59:17.323415 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:17.323368 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:17.323836 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:17.323459 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:17.329420 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:17.329386 2570 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:17.918005 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:17.917976 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6844df6d4c-ndbx2" Sep 16 18:59:17.972856 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:17.972822 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-547d7b6b54-vgl9d"] Sep 16 18:59:23.724177 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:23.724141 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 18:59:35.239782 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.239747 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj"] Sep 16 18:59:35.243045 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.243025 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.245086 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.245059 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a895e-predictor-serving-cert\"" Sep 16 18:59:35.245194 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.245160 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a895e-kube-rbac-proxy-sar-config\"" Sep 16 18:59:35.255019 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.254995 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj"] Sep 16 18:59:35.409185 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.409135 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhbvz\" (UniqueName: \"kubernetes.io/projected/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-api-access-zhbvz\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.409365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.409194 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d258cd9-b127-46e1-ae09-50ddd25c0eff-proxy-tls\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.409365 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.409285 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.509843 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.509737 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.509843 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.509790 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhbvz\" (UniqueName: \"kubernetes.io/projected/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-api-access-zhbvz\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.509843 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.509819 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d258cd9-b127-46e1-ae09-50ddd25c0eff-proxy-tls\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.510518 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.510495 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.512503 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.512483 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d258cd9-b127-46e1-ae09-50ddd25c0eff-proxy-tls\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.519387 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.519355 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhbvz\" (UniqueName: \"kubernetes.io/projected/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-api-access-zhbvz\") pod \"error-404-isvc-a895e-predictor-8c7b557fb-l68lj\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.554155 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.554106 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:35.829829 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:35.829755 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj"] Sep 16 18:59:35.832137 ip-10-0-142-175 kubenswrapper[2570]: W0916 18:59:35.832100 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d258cd9_b127_46e1_ae09_50ddd25c0eff.slice/crio-bc081f85ebd65b0c68fc07722bc78c01f7f4be81933fa8656a439458e7aedea8 WatchSource:0}: Error finding container bc081f85ebd65b0c68fc07722bc78c01f7f4be81933fa8656a439458e7aedea8: Status 404 returned error can't find the container with id bc081f85ebd65b0c68fc07722bc78c01f7f4be81933fa8656a439458e7aedea8 Sep 16 18:59:36.009905 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:36.009835 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" event={"ID":"0d258cd9-b127-46e1-ae09-50ddd25c0eff","Type":"ContainerStarted","Data":"8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d"} Sep 16 18:59:36.010194 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:36.009929 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" event={"ID":"0d258cd9-b127-46e1-ae09-50ddd25c0eff","Type":"ContainerStarted","Data":"298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1"} Sep 16 18:59:36.010194 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:36.009960 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" event={"ID":"0d258cd9-b127-46e1-ae09-50ddd25c0eff","Type":"ContainerStarted","Data":"bc081f85ebd65b0c68fc07722bc78c01f7f4be81933fa8656a439458e7aedea8"} Sep 16 18:59:36.010194 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:36.009998 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:36.029014 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:36.028959 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podStartSLOduration=1.028940563 podStartE2EDuration="1.028940563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:36.027006538 +0000 UTC m=+863.570312297" watchObservedRunningTime="2026-09-16 18:59:36.028940563 +0000 UTC m=+863.572246382" Sep 16 18:59:37.014590 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:37.014557 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:37.016091 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:37.016057 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Sep 16 18:59:38.020307 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:38.020263 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Sep 16 18:59:42.997733 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:42.997684 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-547d7b6b54-vgl9d" podUID="adfd869f-6302-4b33-8128-03c52c0dccef" containerName="console" containerID="cri-o://060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c" gracePeriod=15 Sep 16 18:59:43.025407 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.025375 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 18:59:43.025833 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.025804 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Sep 16 18:59:43.248517 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.248445 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:59:43.384101 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384059 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pppl7\" (UniqueName: \"kubernetes.io/projected/adfd869f-6302-4b33-8128-03c52c0dccef-kube-api-access-pppl7\") pod \"adfd869f-6302-4b33-8128-03c52c0dccef\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " Sep 16 18:59:43.384298 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384128 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-oauth-serving-cert\") pod \"adfd869f-6302-4b33-8128-03c52c0dccef\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " Sep 16 18:59:43.384298 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384150 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-serving-cert\") pod \"adfd869f-6302-4b33-8128-03c52c0dccef\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " Sep 16 18:59:43.384421 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384332 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-trusted-ca-bundle\") pod \"adfd869f-6302-4b33-8128-03c52c0dccef\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " Sep 16 18:59:43.384421 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384402 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-oauth-config\") pod \"adfd869f-6302-4b33-8128-03c52c0dccef\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " Sep 16 18:59:43.384525 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384452 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-console-config\") pod \"adfd869f-6302-4b33-8128-03c52c0dccef\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " Sep 16 18:59:43.384525 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384483 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-service-ca\") pod \"adfd869f-6302-4b33-8128-03c52c0dccef\" (UID: \"adfd869f-6302-4b33-8128-03c52c0dccef\") " Sep 16 18:59:43.384621 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384588 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "adfd869f-6302-4b33-8128-03c52c0dccef" (UID: "adfd869f-6302-4b33-8128-03c52c0dccef"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:43.384753 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384720 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "adfd869f-6302-4b33-8128-03c52c0dccef" (UID: "adfd869f-6302-4b33-8128-03c52c0dccef"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:43.384965 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384946 2570 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-trusted-ca-bundle\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:59:43.385036 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.384970 2570 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-oauth-serving-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:59:43.385162 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.385141 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-service-ca" (OuterVolumeSpecName: "service-ca") pod "adfd869f-6302-4b33-8128-03c52c0dccef" (UID: "adfd869f-6302-4b33-8128-03c52c0dccef"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:43.385200 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.385149 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-console-config" (OuterVolumeSpecName: "console-config") pod "adfd869f-6302-4b33-8128-03c52c0dccef" (UID: "adfd869f-6302-4b33-8128-03c52c0dccef"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:43.386642 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.386613 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfd869f-6302-4b33-8128-03c52c0dccef-kube-api-access-pppl7" (OuterVolumeSpecName: "kube-api-access-pppl7") pod "adfd869f-6302-4b33-8128-03c52c0dccef" (UID: "adfd869f-6302-4b33-8128-03c52c0dccef"). InnerVolumeSpecName "kube-api-access-pppl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:59:43.386759 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.386619 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "adfd869f-6302-4b33-8128-03c52c0dccef" (UID: "adfd869f-6302-4b33-8128-03c52c0dccef"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:59:43.386984 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.386963 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "adfd869f-6302-4b33-8128-03c52c0dccef" (UID: "adfd869f-6302-4b33-8128-03c52c0dccef"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:59:43.485681 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.485644 2570 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-console-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:59:43.485681 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.485674 2570 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/adfd869f-6302-4b33-8128-03c52c0dccef-service-ca\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:59:43.485681 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.485682 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pppl7\" (UniqueName: \"kubernetes.io/projected/adfd869f-6302-4b33-8128-03c52c0dccef-kube-api-access-pppl7\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:59:43.485681 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.485692 2570 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-serving-cert\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:59:43.485974 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:43.485701 2570 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/adfd869f-6302-4b33-8128-03c52c0dccef-console-oauth-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 18:59:44.052305 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.052265 2570 generic.go:356] "Generic (PLEG): container finished" podID="adfd869f-6302-4b33-8128-03c52c0dccef" containerID="060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c" exitCode=0 Sep 16 18:59:44.052718 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.052327 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-547d7b6b54-vgl9d" Sep 16 18:59:44.052718 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.052359 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547d7b6b54-vgl9d" event={"ID":"adfd869f-6302-4b33-8128-03c52c0dccef","Type":"ContainerDied","Data":"060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c"} Sep 16 18:59:44.052718 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.052410 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-547d7b6b54-vgl9d" event={"ID":"adfd869f-6302-4b33-8128-03c52c0dccef","Type":"ContainerDied","Data":"9800d3d8fddb26a00c4187261c54c70e188b27266557ea4b7f8234663709ad6a"} Sep 16 18:59:44.052718 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.052432 2570 scope.go:117] "RemoveContainer" containerID="060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c" Sep 16 18:59:44.061693 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.061671 2570 scope.go:117] "RemoveContainer" containerID="060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c" Sep 16 18:59:44.062065 ip-10-0-142-175 kubenswrapper[2570]: E0916 18:59:44.062037 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c\": container with ID starting with 060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c not found: ID does not exist" containerID="060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c" Sep 16 18:59:44.062146 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.062072 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c"} err="failed to get container status \"060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c\": rpc error: code = NotFound desc = could not find container \"060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c\": container with ID starting with 060a6333f836565170745ba502dca7c00cc28c1946845de23e9534db9af46b4c not found: ID does not exist" Sep 16 18:59:44.080600 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.080565 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-547d7b6b54-vgl9d"] Sep 16 18:59:44.084741 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.084703 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-547d7b6b54-vgl9d"] Sep 16 18:59:44.971880 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:44.971830 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adfd869f-6302-4b33-8128-03c52c0dccef" path="/var/lib/kubelet/pods/adfd869f-6302-4b33-8128-03c52c0dccef/volumes" Sep 16 18:59:53.026465 ip-10-0-142-175 kubenswrapper[2570]: I0916 18:59:53.026423 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Sep 16 19:00:03.026463 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:00:03.026418 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Sep 16 19:00:12.932990 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:00:12.932909 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:00:12.934003 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:00:12.933983 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:00:13.026508 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:00:13.026467 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Sep 16 19:00:23.027209 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:00:23.027177 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 19:05:12.969584 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:05:12.969542 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:05:12.970207 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:05:12.969685 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:07:50.223108 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.223072 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq"] Sep 16 19:07:50.225700 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.223305 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" containerID="cri-o://e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9" gracePeriod=30 Sep 16 19:07:50.225700 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.223367 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kube-rbac-proxy" containerID="cri-o://98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd" gracePeriod=30 Sep 16 19:07:50.345225 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.345187 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7"] Sep 16 19:07:50.345592 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.345580 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adfd869f-6302-4b33-8128-03c52c0dccef" containerName="console" Sep 16 19:07:50.345637 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.345593 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="adfd869f-6302-4b33-8128-03c52c0dccef" containerName="console" Sep 16 19:07:50.345671 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.345658 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="adfd869f-6302-4b33-8128-03c52c0dccef" containerName="console" Sep 16 19:07:50.348774 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.348756 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.351279 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.351253 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-443e6-kube-rbac-proxy-sar-config\"" Sep 16 19:07:50.351462 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.351442 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-443e6-predictor-serving-cert\"" Sep 16 19:07:50.359640 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.359612 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7"] Sep 16 19:07:50.399059 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.399023 2570 generic.go:356] "Generic (PLEG): container finished" podID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerID="98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd" exitCode=2 Sep 16 19:07:50.399218 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.399065 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" event={"ID":"494bb4ca-014f-4713-aff2-6456f8b5a03e","Type":"ContainerDied","Data":"98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd"} Sep 16 19:07:50.488916 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.488796 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ee2ad02-86fe-4710-85b1-bf7131f47883-proxy-tls\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.489147 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.489029 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdg6h\" (UniqueName: \"kubernetes.io/projected/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-api-access-xdg6h\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.489147 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.489065 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.590197 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.590150 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdg6h\" (UniqueName: \"kubernetes.io/projected/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-api-access-xdg6h\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.590392 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.590208 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.590392 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.590228 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ee2ad02-86fe-4710-85b1-bf7131f47883-proxy-tls\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.590932 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.590911 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.593133 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.593072 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ee2ad02-86fe-4710-85b1-bf7131f47883-proxy-tls\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.599969 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.599937 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdg6h\" (UniqueName: \"kubernetes.io/projected/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-api-access-xdg6h\") pod \"error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.659069 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.659034 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:50.936706 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:50.936661 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7"] Sep 16 19:07:50.939051 ip-10-0-142-175 kubenswrapper[2570]: W0916 19:07:50.939026 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee2ad02_86fe_4710_85b1_bf7131f47883.slice/crio-c193789b6fabd46e13defa4ee315aa73916cf147671d196013e6f3084540eaee WatchSource:0}: Error finding container c193789b6fabd46e13defa4ee315aa73916cf147671d196013e6f3084540eaee: Status 404 returned error can't find the container with id c193789b6fabd46e13defa4ee315aa73916cf147671d196013e6f3084540eaee Sep 16 19:07:51.406118 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:51.406084 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" event={"ID":"2ee2ad02-86fe-4710-85b1-bf7131f47883","Type":"ContainerStarted","Data":"eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da"} Sep 16 19:07:51.406535 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:51.406143 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" event={"ID":"2ee2ad02-86fe-4710-85b1-bf7131f47883","Type":"ContainerStarted","Data":"aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f"} Sep 16 19:07:51.406535 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:51.406212 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:51.406535 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:51.406242 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" event={"ID":"2ee2ad02-86fe-4710-85b1-bf7131f47883","Type":"ContainerStarted","Data":"c193789b6fabd46e13defa4ee315aa73916cf147671d196013e6f3084540eaee"} Sep 16 19:07:51.406535 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:51.406269 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:51.407973 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:51.407943 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:07:51.425006 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:51.424959 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podStartSLOduration=1.424946706 podStartE2EDuration="1.424946706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:51.423412229 +0000 UTC m=+1358.966717987" watchObservedRunningTime="2026-09-16 19:07:51.424946706 +0000 UTC m=+1358.968252465" Sep 16 19:07:52.412177 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:52.412139 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:07:53.675965 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.675935 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 19:07:53.818076 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.817987 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls\") pod \"494bb4ca-014f-4713-aff2-6456f8b5a03e\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " Sep 16 19:07:53.818076 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.818044 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkdtn\" (UniqueName: \"kubernetes.io/projected/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-api-access-tkdtn\") pod \"494bb4ca-014f-4713-aff2-6456f8b5a03e\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " Sep 16 19:07:53.818387 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.818104 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-rbac-proxy-sar-config\") pod \"494bb4ca-014f-4713-aff2-6456f8b5a03e\" (UID: \"494bb4ca-014f-4713-aff2-6456f8b5a03e\") " Sep 16 19:07:53.818490 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.818466 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "494bb4ca-014f-4713-aff2-6456f8b5a03e" (UID: "494bb4ca-014f-4713-aff2-6456f8b5a03e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:53.820291 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.820264 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "494bb4ca-014f-4713-aff2-6456f8b5a03e" (UID: "494bb4ca-014f-4713-aff2-6456f8b5a03e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:07:53.820392 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.820347 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-api-access-tkdtn" (OuterVolumeSpecName: "kube-api-access-tkdtn") pod "494bb4ca-014f-4713-aff2-6456f8b5a03e" (UID: "494bb4ca-014f-4713-aff2-6456f8b5a03e"). InnerVolumeSpecName "kube-api-access-tkdtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:07:53.919402 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.919351 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/494bb4ca-014f-4713-aff2-6456f8b5a03e-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:07:53.919402 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.919394 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkdtn\" (UniqueName: \"kubernetes.io/projected/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-api-access-tkdtn\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:07:53.919402 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:53.919405 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/494bb4ca-014f-4713-aff2-6456f8b5a03e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:07:54.423343 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.423308 2570 generic.go:356] "Generic (PLEG): container finished" podID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerID="e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9" exitCode=0 Sep 16 19:07:54.423514 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.423360 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" Sep 16 19:07:54.423514 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.423379 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" event={"ID":"494bb4ca-014f-4713-aff2-6456f8b5a03e","Type":"ContainerDied","Data":"e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9"} Sep 16 19:07:54.423514 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.423427 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq" event={"ID":"494bb4ca-014f-4713-aff2-6456f8b5a03e","Type":"ContainerDied","Data":"bd88ff2dee4afd56bc596c56d771b1081ff6a47b6c2799d33b3e3eec2dad5fc4"} Sep 16 19:07:54.423514 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.423452 2570 scope.go:117] "RemoveContainer" containerID="98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd" Sep 16 19:07:54.433020 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.432947 2570 scope.go:117] "RemoveContainer" containerID="e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9" Sep 16 19:07:54.440644 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.440623 2570 scope.go:117] "RemoveContainer" containerID="98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd" Sep 16 19:07:54.440931 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:07:54.440913 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd\": container with ID starting with 98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd not found: ID does not exist" containerID="98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd" Sep 16 19:07:54.441007 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.440936 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd"} err="failed to get container status \"98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd\": rpc error: code = NotFound desc = could not find container \"98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd\": container with ID starting with 98f6ed8d6e4970b32a2b57dc2e9b61bdc6e0b2f7cd646b07e1afac9af4312dcd not found: ID does not exist" Sep 16 19:07:54.441007 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.440952 2570 scope.go:117] "RemoveContainer" containerID="e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9" Sep 16 19:07:54.441191 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:07:54.441173 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9\": container with ID starting with e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9 not found: ID does not exist" containerID="e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9" Sep 16 19:07:54.441233 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.441197 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9"} err="failed to get container status \"e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9\": rpc error: code = NotFound desc = could not find container \"e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9\": container with ID starting with e3fc4d0fa083f07fe8c9cbc99bb0a4059c022a248494196e1ad9c3e0ac30bbf9 not found: ID does not exist" Sep 16 19:07:54.444258 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.444237 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq"] Sep 16 19:07:54.448257 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.448238 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9a5c2-predictor-6999f87ffb-m76lq"] Sep 16 19:07:54.971315 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:54.971283 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" path="/var/lib/kubelet/pods/494bb4ca-014f-4713-aff2-6456f8b5a03e/volumes" Sep 16 19:07:57.417419 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:57.417391 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:07:57.417933 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:07:57.417907 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:08:07.418823 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:07.418783 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:08:17.418065 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:17.418013 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:08:27.418182 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:27.418137 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused" Sep 16 19:08:37.419716 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:37.419681 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:08:49.871696 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:49.871657 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj"] Sep 16 19:08:49.872155 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:49.871936 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" containerID="cri-o://298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1" gracePeriod=30 Sep 16 19:08:49.872155 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:49.871991 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kube-rbac-proxy" containerID="cri-o://8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d" gracePeriod=30 Sep 16 19:08:50.068661 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.068625 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628"] Sep 16 19:08:50.069049 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.069035 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kube-rbac-proxy" Sep 16 19:08:50.069106 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.069051 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kube-rbac-proxy" Sep 16 19:08:50.069106 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.069073 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" Sep 16 19:08:50.069106 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.069078 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" Sep 16 19:08:50.069205 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.069141 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kube-rbac-proxy" Sep 16 19:08:50.069205 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.069151 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="494bb4ca-014f-4713-aff2-6456f8b5a03e" containerName="kserve-container" Sep 16 19:08:50.072389 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.072372 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.075168 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.075144 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9450b-kube-rbac-proxy-sar-config\"" Sep 16 19:08:50.075296 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.075219 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-9450b-predictor-serving-cert\"" Sep 16 19:08:50.083309 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.083283 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628"] Sep 16 19:08:50.213179 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.213091 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nn9z\" (UniqueName: \"kubernetes.io/projected/188005de-15e8-4854-936b-6e27acfaf44b-kube-api-access-6nn9z\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.213179 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.213127 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188005de-15e8-4854-936b-6e27acfaf44b-proxy-tls\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.213412 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.213249 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188005de-15e8-4854-936b-6e27acfaf44b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.313975 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.313934 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188005de-15e8-4854-936b-6e27acfaf44b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.314182 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.314011 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nn9z\" (UniqueName: \"kubernetes.io/projected/188005de-15e8-4854-936b-6e27acfaf44b-kube-api-access-6nn9z\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.314182 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.314046 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188005de-15e8-4854-936b-6e27acfaf44b-proxy-tls\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.314614 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.314583 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188005de-15e8-4854-936b-6e27acfaf44b-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.316703 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.316679 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188005de-15e8-4854-936b-6e27acfaf44b-proxy-tls\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.325709 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.325648 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nn9z\" (UniqueName: \"kubernetes.io/projected/188005de-15e8-4854-936b-6e27acfaf44b-kube-api-access-6nn9z\") pod \"error-404-isvc-9450b-predictor-59464df8d8-nx628\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.384206 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.384160 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:50.662219 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.662193 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628"] Sep 16 19:08:50.664835 ip-10-0-142-175 kubenswrapper[2570]: W0916 19:08:50.664797 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188005de_15e8_4854_936b_6e27acfaf44b.slice/crio-af30a1d887286794b1079ba4664b716934ec43f82f6ad32cf4aedf4e6fde2847 WatchSource:0}: Error finding container af30a1d887286794b1079ba4664b716934ec43f82f6ad32cf4aedf4e6fde2847: Status 404 returned error can't find the container with id af30a1d887286794b1079ba4664b716934ec43f82f6ad32cf4aedf4e6fde2847 Sep 16 19:08:50.693235 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.693207 2570 generic.go:356] "Generic (PLEG): container finished" podID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerID="8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d" exitCode=2 Sep 16 19:08:50.693374 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.693288 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" event={"ID":"0d258cd9-b127-46e1-ae09-50ddd25c0eff","Type":"ContainerDied","Data":"8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d"} Sep 16 19:08:50.694527 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:50.694507 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" event={"ID":"188005de-15e8-4854-936b-6e27acfaf44b","Type":"ContainerStarted","Data":"af30a1d887286794b1079ba4664b716934ec43f82f6ad32cf4aedf4e6fde2847"} Sep 16 19:08:51.700630 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:51.700591 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" event={"ID":"188005de-15e8-4854-936b-6e27acfaf44b","Type":"ContainerStarted","Data":"2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c"} Sep 16 19:08:51.701097 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:51.700650 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" event={"ID":"188005de-15e8-4854-936b-6e27acfaf44b","Type":"ContainerStarted","Data":"d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657"} Sep 16 19:08:51.701097 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:51.700676 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:51.701097 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:51.700725 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:51.702289 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:51.702263 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:08:51.719616 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:51.719566 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podStartSLOduration=1.719548639 podStartE2EDuration="1.719548639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:51.71695244 +0000 UTC m=+1419.260258198" watchObservedRunningTime="2026-09-16 19:08:51.719548639 +0000 UTC m=+1419.262854398" Sep 16 19:08:52.706012 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:52.705976 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:08:53.021382 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.021347 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused" Sep 16 19:08:53.026645 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.026615 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused" Sep 16 19:08:53.231263 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.231234 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 19:08:53.242414 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.242351 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-rbac-proxy-sar-config\") pod \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " Sep 16 19:08:53.242536 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.242454 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d258cd9-b127-46e1-ae09-50ddd25c0eff-proxy-tls\") pod \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " Sep 16 19:08:53.242536 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.242484 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhbvz\" (UniqueName: \"kubernetes.io/projected/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-api-access-zhbvz\") pod \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\" (UID: \"0d258cd9-b127-46e1-ae09-50ddd25c0eff\") " Sep 16 19:08:53.242650 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.242613 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0d258cd9-b127-46e1-ae09-50ddd25c0eff" (UID: "0d258cd9-b127-46e1-ae09-50ddd25c0eff"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:08:53.242732 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.242717 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:08:53.244538 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.244512 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-api-access-zhbvz" (OuterVolumeSpecName: "kube-api-access-zhbvz") pod "0d258cd9-b127-46e1-ae09-50ddd25c0eff" (UID: "0d258cd9-b127-46e1-ae09-50ddd25c0eff"). InnerVolumeSpecName "kube-api-access-zhbvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:08:53.244628 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.244610 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d258cd9-b127-46e1-ae09-50ddd25c0eff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0d258cd9-b127-46e1-ae09-50ddd25c0eff" (UID: "0d258cd9-b127-46e1-ae09-50ddd25c0eff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:08:53.343554 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.343521 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0d258cd9-b127-46e1-ae09-50ddd25c0eff-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:08:53.343554 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.343547 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhbvz\" (UniqueName: \"kubernetes.io/projected/0d258cd9-b127-46e1-ae09-50ddd25c0eff-kube-api-access-zhbvz\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:08:53.712337 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.712254 2570 generic.go:356] "Generic (PLEG): container finished" podID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerID="298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1" exitCode=0 Sep 16 19:08:53.712337 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.712306 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" Sep 16 19:08:53.712774 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.712309 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" event={"ID":"0d258cd9-b127-46e1-ae09-50ddd25c0eff","Type":"ContainerDied","Data":"298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1"} Sep 16 19:08:53.712774 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.712452 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj" event={"ID":"0d258cd9-b127-46e1-ae09-50ddd25c0eff","Type":"ContainerDied","Data":"bc081f85ebd65b0c68fc07722bc78c01f7f4be81933fa8656a439458e7aedea8"} Sep 16 19:08:53.712774 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.712490 2570 scope.go:117] "RemoveContainer" containerID="8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d" Sep 16 19:08:53.721734 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.721718 2570 scope.go:117] "RemoveContainer" containerID="298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1" Sep 16 19:08:53.730076 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.730054 2570 scope.go:117] "RemoveContainer" containerID="8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d" Sep 16 19:08:53.730371 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:08:53.730352 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d\": container with ID starting with 8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d not found: ID does not exist" containerID="8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d" Sep 16 19:08:53.730430 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.730380 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d"} err="failed to get container status \"8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d\": rpc error: code = NotFound desc = could not find container \"8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d\": container with ID starting with 8dc9c93dff0e2ef648577608155e156eeee320aac3b2317d8e7708285eb9020d not found: ID does not exist" Sep 16 19:08:53.730430 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.730398 2570 scope.go:117] "RemoveContainer" containerID="298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1" Sep 16 19:08:53.730661 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:08:53.730646 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1\": container with ID starting with 298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1 not found: ID does not exist" containerID="298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1" Sep 16 19:08:53.730698 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.730667 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1"} err="failed to get container status \"298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1\": rpc error: code = NotFound desc = could not find container \"298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1\": container with ID starting with 298395d9e9deb462bea8b3953e4a605e6f04a10453b19372a6e0b1fcd699dcf1 not found: ID does not exist" Sep 16 19:08:53.735328 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.735304 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj"] Sep 16 19:08:53.737012 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:53.736990 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a895e-predictor-8c7b557fb-l68lj"] Sep 16 19:08:54.972759 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:54.972725 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" path="/var/lib/kubelet/pods/0d258cd9-b127-46e1-ae09-50ddd25c0eff/volumes" Sep 16 19:08:57.710785 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:57.710757 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:08:57.711230 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:08:57.711208 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:09:00.395156 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.395117 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7"] Sep 16 19:09:00.395672 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.395367 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" containerID="cri-o://aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f" gracePeriod=30 Sep 16 19:09:00.395672 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.395459 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kube-rbac-proxy" containerID="cri-o://eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da" gracePeriod=30 Sep 16 19:09:00.750021 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.749988 2570 generic.go:356] "Generic (PLEG): container finished" podID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerID="eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da" exitCode=2 Sep 16 19:09:00.750224 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.750073 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" event={"ID":"2ee2ad02-86fe-4710-85b1-bf7131f47883","Type":"ContainerDied","Data":"eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da"} Sep 16 19:09:00.775297 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.775261 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9"] Sep 16 19:09:00.775668 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.775655 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" Sep 16 19:09:00.775714 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.775670 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" Sep 16 19:09:00.775714 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.775691 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kube-rbac-proxy" Sep 16 19:09:00.775714 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.775696 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kube-rbac-proxy" Sep 16 19:09:00.775812 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.775755 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kube-rbac-proxy" Sep 16 19:09:00.775812 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.775761 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d258cd9-b127-46e1-ae09-50ddd25c0eff" containerName="kserve-container" Sep 16 19:09:00.778732 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.778716 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.784186 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.784158 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-27911-predictor-serving-cert\"" Sep 16 19:09:00.784361 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.784184 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-27911-kube-rbac-proxy-sar-config\"" Sep 16 19:09:00.798928 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.798846 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.799103 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.798969 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-proxy-tls\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.799103 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.799014 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwc8\" (UniqueName: \"kubernetes.io/projected/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-api-access-gcwc8\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.838075 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.838029 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9"] Sep 16 19:09:00.900121 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.900082 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.900121 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.900130 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-proxy-tls\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.900358 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.900171 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwc8\" (UniqueName: \"kubernetes.io/projected/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-api-access-gcwc8\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.900777 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.900756 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.902921 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.902897 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-proxy-tls\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:00.908547 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:00.908518 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwc8\" (UniqueName: \"kubernetes.io/projected/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-api-access-gcwc8\") pod \"error-404-isvc-27911-predictor-6965d7c55b-x57m9\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:01.090650 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:01.090555 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:01.372300 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:01.372272 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9"] Sep 16 19:09:01.375022 ip-10-0-142-175 kubenswrapper[2570]: W0916 19:09:01.374995 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8777970_6d1f_4eac_b74e_8a740e4e0aa4.slice/crio-33737ab5103635a250428dd1395552645dc6362b135c1ec87344f2a8d51b20c0 WatchSource:0}: Error finding container 33737ab5103635a250428dd1395552645dc6362b135c1ec87344f2a8d51b20c0: Status 404 returned error can't find the container with id 33737ab5103635a250428dd1395552645dc6362b135c1ec87344f2a8d51b20c0 Sep 16 19:09:01.757081 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:01.757041 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" event={"ID":"d8777970-6d1f-4eac-b74e-8a740e4e0aa4","Type":"ContainerStarted","Data":"5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf"} Sep 16 19:09:01.757493 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:01.757113 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" event={"ID":"d8777970-6d1f-4eac-b74e-8a740e4e0aa4","Type":"ContainerStarted","Data":"b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373"} Sep 16 19:09:01.757493 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:01.757144 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" event={"ID":"d8777970-6d1f-4eac-b74e-8a740e4e0aa4","Type":"ContainerStarted","Data":"33737ab5103635a250428dd1395552645dc6362b135c1ec87344f2a8d51b20c0"} Sep 16 19:09:01.757493 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:01.757174 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:01.777141 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:01.777085 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podStartSLOduration=1.7770627669999999 podStartE2EDuration="1.777062767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:09:01.774635248 +0000 UTC m=+1429.317941018" watchObservedRunningTime="2026-09-16 19:09:01.777062767 +0000 UTC m=+1429.320368526" Sep 16 19:09:02.412731 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:02.412682 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused" Sep 16 19:09:02.761981 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:02.761939 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:02.763210 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:02.763185 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:09:03.766658 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:03.766565 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:09:03.945203 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:03.945177 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:09:04.028259 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.028171 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdg6h\" (UniqueName: \"kubernetes.io/projected/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-api-access-xdg6h\") pod \"2ee2ad02-86fe-4710-85b1-bf7131f47883\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " Sep 16 19:09:04.028435 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.028264 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ee2ad02-86fe-4710-85b1-bf7131f47883-proxy-tls\") pod \"2ee2ad02-86fe-4710-85b1-bf7131f47883\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " Sep 16 19:09:04.028435 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.028292 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-rbac-proxy-sar-config\") pod \"2ee2ad02-86fe-4710-85b1-bf7131f47883\" (UID: \"2ee2ad02-86fe-4710-85b1-bf7131f47883\") " Sep 16 19:09:04.028672 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.028646 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ee2ad02-86fe-4710-85b1-bf7131f47883" (UID: "2ee2ad02-86fe-4710-85b1-bf7131f47883"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:09:04.030382 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.030358 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-api-access-xdg6h" (OuterVolumeSpecName: "kube-api-access-xdg6h") pod "2ee2ad02-86fe-4710-85b1-bf7131f47883" (UID: "2ee2ad02-86fe-4710-85b1-bf7131f47883"). InnerVolumeSpecName "kube-api-access-xdg6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:09:04.030535 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.030505 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ee2ad02-86fe-4710-85b1-bf7131f47883-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ee2ad02-86fe-4710-85b1-bf7131f47883" (UID: "2ee2ad02-86fe-4710-85b1-bf7131f47883"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:09:04.128846 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.128807 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ee2ad02-86fe-4710-85b1-bf7131f47883-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:09:04.128846 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.128837 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:09:04.128846 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.128846 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdg6h\" (UniqueName: \"kubernetes.io/projected/2ee2ad02-86fe-4710-85b1-bf7131f47883-kube-api-access-xdg6h\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:09:04.773101 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.773062 2570 generic.go:356] "Generic (PLEG): container finished" podID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerID="aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f" exitCode=0 Sep 16 19:09:04.773490 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.773118 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" Sep 16 19:09:04.773490 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.773145 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" event={"ID":"2ee2ad02-86fe-4710-85b1-bf7131f47883","Type":"ContainerDied","Data":"aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f"} Sep 16 19:09:04.773490 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.773208 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7" event={"ID":"2ee2ad02-86fe-4710-85b1-bf7131f47883","Type":"ContainerDied","Data":"c193789b6fabd46e13defa4ee315aa73916cf147671d196013e6f3084540eaee"} Sep 16 19:09:04.773490 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.773236 2570 scope.go:117] "RemoveContainer" containerID="eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da" Sep 16 19:09:04.783954 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.783933 2570 scope.go:117] "RemoveContainer" containerID="aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f" Sep 16 19:09:04.791786 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.791766 2570 scope.go:117] "RemoveContainer" containerID="eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da" Sep 16 19:09:04.792076 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:09:04.792060 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da\": container with ID starting with eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da not found: ID does not exist" containerID="eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da" Sep 16 19:09:04.792126 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.792084 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da"} err="failed to get container status \"eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da\": rpc error: code = NotFound desc = could not find container \"eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da\": container with ID starting with eada079a62c5878fd7ff439e3b938dadd44472466c3f71029226d63a783975da not found: ID does not exist" Sep 16 19:09:04.792126 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.792101 2570 scope.go:117] "RemoveContainer" containerID="aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f" Sep 16 19:09:04.792346 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:09:04.792325 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f\": container with ID starting with aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f not found: ID does not exist" containerID="aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f" Sep 16 19:09:04.792394 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.792355 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f"} err="failed to get container status \"aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f\": rpc error: code = NotFound desc = could not find container \"aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f\": container with ID starting with aeb438c71101cd8b67e11095533a43647ba8ff986172ec9e891586341466532f not found: ID does not exist" Sep 16 19:09:04.795120 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.795098 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7"] Sep 16 19:09:04.799197 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.799180 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-443e6-predictor-569c8dcb6d-4kpx7"] Sep 16 19:09:04.971666 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:04.971628 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" path="/var/lib/kubelet/pods/2ee2ad02-86fe-4710-85b1-bf7131f47883/volumes" Sep 16 19:09:07.711962 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:07.711922 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:09:08.771856 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:08.771828 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:09:08.772361 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:08.772334 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:09:17.712194 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:17.712156 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:09:18.772794 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:18.772751 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:09:27.711588 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:27.711545 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused" Sep 16 19:09:28.772309 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:28.772273 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:09:37.711801 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:37.711768 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:09:38.772584 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:38.772544 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:09:48.773880 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:09:48.773840 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:10:00.148800 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.148705 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628"] Sep 16 19:10:00.149232 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.148958 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" containerID="cri-o://d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657" gracePeriod=30 Sep 16 19:10:00.149232 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.149025 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kube-rbac-proxy" containerID="cri-o://2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c" gracePeriod=30 Sep 16 19:10:00.408481 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.408399 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8"] Sep 16 19:10:00.408799 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.408787 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" Sep 16 19:10:00.408843 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.408801 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" Sep 16 19:10:00.408843 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.408809 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kube-rbac-proxy" Sep 16 19:10:00.408843 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.408815 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kube-rbac-proxy" Sep 16 19:10:00.408970 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.408881 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kube-rbac-proxy" Sep 16 19:10:00.408970 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.408893 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ee2ad02-86fe-4710-85b1-bf7131f47883" containerName="kserve-container" Sep 16 19:10:00.412114 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.412096 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.414595 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.414575 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53a10-kube-rbac-proxy-sar-config\"" Sep 16 19:10:00.414595 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.414592 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-53a10-predictor-serving-cert\"" Sep 16 19:10:00.423272 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.423251 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8"] Sep 16 19:10:00.535890 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.535835 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmzqr\" (UniqueName: \"kubernetes.io/projected/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-api-access-gmzqr\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.535890 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.535894 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.536113 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.536027 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24533bc-9ebc-4233-8a25-a5d1f42c77de-proxy-tls\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.636982 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.636952 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24533bc-9ebc-4233-8a25-a5d1f42c77de-proxy-tls\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.637149 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.636995 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmzqr\" (UniqueName: \"kubernetes.io/projected/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-api-access-gmzqr\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.637149 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.637027 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.637684 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.637659 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.639674 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.639649 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24533bc-9ebc-4233-8a25-a5d1f42c77de-proxy-tls\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.644710 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.644689 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmzqr\" (UniqueName: \"kubernetes.io/projected/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-api-access-gmzqr\") pod \"error-404-isvc-53a10-predictor-765f8bbccb-v7px8\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.722787 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.722758 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:00.998704 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:00.998649 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8"] Sep 16 19:10:01.001063 ip-10-0-142-175 kubenswrapper[2570]: W0916 19:10:01.001026 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda24533bc_9ebc_4233_8a25_a5d1f42c77de.slice/crio-5fb3373102f4b34fa70f65a11f4ec36217f954287cba12a8325ac3b4ee39db12 WatchSource:0}: Error finding container 5fb3373102f4b34fa70f65a11f4ec36217f954287cba12a8325ac3b4ee39db12: Status 404 returned error can't find the container with id 5fb3373102f4b34fa70f65a11f4ec36217f954287cba12a8325ac3b4ee39db12 Sep 16 19:10:01.055203 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:01.055171 2570 generic.go:356] "Generic (PLEG): container finished" podID="188005de-15e8-4854-936b-6e27acfaf44b" containerID="2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c" exitCode=2 Sep 16 19:10:01.055398 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:01.055250 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" event={"ID":"188005de-15e8-4854-936b-6e27acfaf44b","Type":"ContainerDied","Data":"2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c"} Sep 16 19:10:01.056380 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:01.056358 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" event={"ID":"a24533bc-9ebc-4233-8a25-a5d1f42c77de","Type":"ContainerStarted","Data":"5fb3373102f4b34fa70f65a11f4ec36217f954287cba12a8325ac3b4ee39db12"} Sep 16 19:10:02.064153 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:02.064116 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" event={"ID":"a24533bc-9ebc-4233-8a25-a5d1f42c77de","Type":"ContainerStarted","Data":"5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63"} Sep 16 19:10:02.064685 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:02.064179 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" event={"ID":"a24533bc-9ebc-4233-8a25-a5d1f42c77de","Type":"ContainerStarted","Data":"4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0"} Sep 16 19:10:02.064685 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:02.064204 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:02.064685 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:02.064317 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:02.065804 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:02.065779 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:02.086787 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:02.086721 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podStartSLOduration=2.08670491 podStartE2EDuration="2.08670491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:10:02.085234017 +0000 UTC m=+1489.628539778" watchObservedRunningTime="2026-09-16 19:10:02.08670491 +0000 UTC m=+1489.630010668" Sep 16 19:10:02.706526 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:02.706479 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused" Sep 16 19:10:03.069111 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.069073 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:03.706911 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.706887 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:10:03.766823 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.766720 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188005de-15e8-4854-936b-6e27acfaf44b-kube-rbac-proxy-sar-config\") pod \"188005de-15e8-4854-936b-6e27acfaf44b\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " Sep 16 19:10:03.767029 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.766891 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188005de-15e8-4854-936b-6e27acfaf44b-proxy-tls\") pod \"188005de-15e8-4854-936b-6e27acfaf44b\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " Sep 16 19:10:03.767029 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.766961 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nn9z\" (UniqueName: \"kubernetes.io/projected/188005de-15e8-4854-936b-6e27acfaf44b-kube-api-access-6nn9z\") pod \"188005de-15e8-4854-936b-6e27acfaf44b\" (UID: \"188005de-15e8-4854-936b-6e27acfaf44b\") " Sep 16 19:10:03.767142 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.767103 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/188005de-15e8-4854-936b-6e27acfaf44b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "188005de-15e8-4854-936b-6e27acfaf44b" (UID: "188005de-15e8-4854-936b-6e27acfaf44b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:10:03.767234 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.767220 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/188005de-15e8-4854-936b-6e27acfaf44b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:10:03.769323 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.769294 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/188005de-15e8-4854-936b-6e27acfaf44b-kube-api-access-6nn9z" (OuterVolumeSpecName: "kube-api-access-6nn9z") pod "188005de-15e8-4854-936b-6e27acfaf44b" (UID: "188005de-15e8-4854-936b-6e27acfaf44b"). InnerVolumeSpecName "kube-api-access-6nn9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:10:03.769455 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.769294 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/188005de-15e8-4854-936b-6e27acfaf44b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "188005de-15e8-4854-936b-6e27acfaf44b" (UID: "188005de-15e8-4854-936b-6e27acfaf44b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:10:03.868198 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.868160 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/188005de-15e8-4854-936b-6e27acfaf44b-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:10:03.868198 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:03.868190 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nn9z\" (UniqueName: \"kubernetes.io/projected/188005de-15e8-4854-936b-6e27acfaf44b-kube-api-access-6nn9z\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:10:04.074933 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.074828 2570 generic.go:356] "Generic (PLEG): container finished" podID="188005de-15e8-4854-936b-6e27acfaf44b" containerID="d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657" exitCode=0 Sep 16 19:10:04.074933 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.074900 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" Sep 16 19:10:04.075425 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.074903 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" event={"ID":"188005de-15e8-4854-936b-6e27acfaf44b","Type":"ContainerDied","Data":"d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657"} Sep 16 19:10:04.075425 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.075040 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628" event={"ID":"188005de-15e8-4854-936b-6e27acfaf44b","Type":"ContainerDied","Data":"af30a1d887286794b1079ba4664b716934ec43f82f6ad32cf4aedf4e6fde2847"} Sep 16 19:10:04.075425 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.075086 2570 scope.go:117] "RemoveContainer" containerID="2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c" Sep 16 19:10:04.084298 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.084276 2570 scope.go:117] "RemoveContainer" containerID="d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657" Sep 16 19:10:04.092459 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.092438 2570 scope.go:117] "RemoveContainer" containerID="2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c" Sep 16 19:10:04.092778 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:10:04.092751 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c\": container with ID starting with 2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c not found: ID does not exist" containerID="2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c" Sep 16 19:10:04.092847 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.092780 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c"} err="failed to get container status \"2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c\": rpc error: code = NotFound desc = could not find container \"2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c\": container with ID starting with 2623786a2a535cd08cbafdda658b5e4d088c38bebce2d3eeb677142412fe953c not found: ID does not exist" Sep 16 19:10:04.092847 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.092799 2570 scope.go:117] "RemoveContainer" containerID="d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657" Sep 16 19:10:04.093076 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:10:04.093058 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657\": container with ID starting with d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657 not found: ID does not exist" containerID="d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657" Sep 16 19:10:04.093127 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.093083 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657"} err="failed to get container status \"d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657\": rpc error: code = NotFound desc = could not find container \"d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657\": container with ID starting with d2552bc20e04483790642db18dad3e4871e26d6355059d010f9986855788b657 not found: ID does not exist" Sep 16 19:10:04.100187 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.100165 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628"] Sep 16 19:10:04.107322 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.107303 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-9450b-predictor-59464df8d8-nx628"] Sep 16 19:10:04.970823 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:04.970791 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="188005de-15e8-4854-936b-6e27acfaf44b" path="/var/lib/kubelet/pods/188005de-15e8-4854-936b-6e27acfaf44b/volumes" Sep 16 19:10:08.073930 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:08.073900 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:10:08.074455 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:08.074430 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:13.011885 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:13.011834 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:10:13.013395 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:13.013370 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:10:18.075165 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:18.075125 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:28.075178 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:28.075137 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:38.075433 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:38.075390 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:10:48.075837 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:10:48.075805 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:15:13.049110 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:15:13.049080 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:15:13.051019 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:15:13.050994 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:18:15.516305 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.516229 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9"] Sep 16 19:18:15.518897 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.516508 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" containerID="cri-o://b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373" gracePeriod=30 Sep 16 19:18:15.518897 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.516559 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kube-rbac-proxy" containerID="cri-o://5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf" gracePeriod=30 Sep 16 19:18:15.687152 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.687115 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g"] Sep 16 19:18:15.687523 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.687508 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" Sep 16 19:18:15.687523 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.687522 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" Sep 16 19:18:15.687678 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.687530 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kube-rbac-proxy" Sep 16 19:18:15.687678 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.687535 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kube-rbac-proxy" Sep 16 19:18:15.687678 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.687600 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kube-rbac-proxy" Sep 16 19:18:15.687678 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.687611 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="188005de-15e8-4854-936b-6e27acfaf44b" containerName="kserve-container" Sep 16 19:18:15.690662 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.690642 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.692989 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.692969 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3bba6-predictor-serving-cert\"" Sep 16 19:18:15.693101 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.693015 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-3bba6-kube-rbac-proxy-sar-config\"" Sep 16 19:18:15.700541 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.700515 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g"] Sep 16 19:18:15.718622 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.718583 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-proxy-tls\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.718804 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.718650 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-api-access-ct595\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.718804 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.718781 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.819602 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.819511 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-proxy-tls\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.819602 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.819576 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-api-access-ct595\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.819845 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.819627 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.820286 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.820267 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.822210 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.822190 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-proxy-tls\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:15.827825 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:15.827798 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-api-access-ct595\") pod \"error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:16.001983 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.001944 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:16.276928 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.276905 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g"] Sep 16 19:18:16.279268 ip-10-0-142-175 kubenswrapper[2570]: W0916 19:18:16.279240 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdbc5b8_e91e_42fe_b995_3b89ce0638b1.slice/crio-3991b6fde0de85423220faa139b56084bc5c53df0e0307afc334c2791accb87a WatchSource:0}: Error finding container 3991b6fde0de85423220faa139b56084bc5c53df0e0307afc334c2791accb87a: Status 404 returned error can't find the container with id 3991b6fde0de85423220faa139b56084bc5c53df0e0307afc334c2791accb87a Sep 16 19:18:16.483251 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.483200 2570 generic.go:356] "Generic (PLEG): container finished" podID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerID="5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf" exitCode=2 Sep 16 19:18:16.483423 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.483276 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" event={"ID":"d8777970-6d1f-4eac-b74e-8a740e4e0aa4","Type":"ContainerDied","Data":"5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf"} Sep 16 19:18:16.485925 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.485895 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" event={"ID":"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1","Type":"ContainerStarted","Data":"b90c4c23dff22e50836c20a7d5400b48e5df370ba7ad50d8a6311d482aea5568"} Sep 16 19:18:16.486092 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.485953 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" event={"ID":"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1","Type":"ContainerStarted","Data":"6280326b8bd4c04374b6321de22a7214db850c038467c4f9d941f44f986a7d94"} Sep 16 19:18:16.486092 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.485991 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:16.486092 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.486018 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" event={"ID":"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1","Type":"ContainerStarted","Data":"3991b6fde0de85423220faa139b56084bc5c53df0e0307afc334c2791accb87a"} Sep 16 19:18:16.506675 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:16.506623 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podStartSLOduration=1.506607354 podStartE2EDuration="1.506607354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:16.505092436 +0000 UTC m=+1984.048398194" watchObservedRunningTime="2026-09-16 19:18:16.506607354 +0000 UTC m=+1984.049913113" Sep 16 19:18:17.491326 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:17.491294 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:17.493000 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:17.492968 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:18:18.496332 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:18.496293 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:18:18.767695 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:18.767598 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused" Sep 16 19:18:18.773037 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:18.773006 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused" Sep 16 19:18:18.977296 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:18.977266 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:18:19.046789 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.046690 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-proxy-tls\") pod \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " Sep 16 19:18:19.046789 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.046774 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-rbac-proxy-sar-config\") pod \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " Sep 16 19:18:19.047063 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.046807 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcwc8\" (UniqueName: \"kubernetes.io/projected/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-api-access-gcwc8\") pod \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\" (UID: \"d8777970-6d1f-4eac-b74e-8a740e4e0aa4\") " Sep 16 19:18:19.047211 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.047185 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8777970-6d1f-4eac-b74e-8a740e4e0aa4" (UID: "d8777970-6d1f-4eac-b74e-8a740e4e0aa4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:18:19.049005 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.048976 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8777970-6d1f-4eac-b74e-8a740e4e0aa4" (UID: "d8777970-6d1f-4eac-b74e-8a740e4e0aa4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:18:19.049091 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.049030 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-api-access-gcwc8" (OuterVolumeSpecName: "kube-api-access-gcwc8") pod "d8777970-6d1f-4eac-b74e-8a740e4e0aa4" (UID: "d8777970-6d1f-4eac-b74e-8a740e4e0aa4"). InnerVolumeSpecName "kube-api-access-gcwc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:18:19.147992 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.147841 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:18:19.147992 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.147985 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:18:19.147992 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.147997 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwc8\" (UniqueName: \"kubernetes.io/projected/d8777970-6d1f-4eac-b74e-8a740e4e0aa4-kube-api-access-gcwc8\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:18:19.502745 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.502711 2570 generic.go:356] "Generic (PLEG): container finished" podID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerID="b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373" exitCode=0 Sep 16 19:18:19.503167 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.502762 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" Sep 16 19:18:19.503167 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.502764 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" event={"ID":"d8777970-6d1f-4eac-b74e-8a740e4e0aa4","Type":"ContainerDied","Data":"b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373"} Sep 16 19:18:19.503167 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.502914 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9" event={"ID":"d8777970-6d1f-4eac-b74e-8a740e4e0aa4","Type":"ContainerDied","Data":"33737ab5103635a250428dd1395552645dc6362b135c1ec87344f2a8d51b20c0"} Sep 16 19:18:19.503167 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.502949 2570 scope.go:117] "RemoveContainer" containerID="5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf" Sep 16 19:18:19.512227 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.512210 2570 scope.go:117] "RemoveContainer" containerID="b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373" Sep 16 19:18:19.520750 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.520729 2570 scope.go:117] "RemoveContainer" containerID="5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf" Sep 16 19:18:19.521071 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:18:19.521052 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf\": container with ID starting with 5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf not found: ID does not exist" containerID="5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf" Sep 16 19:18:19.521139 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.521083 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf"} err="failed to get container status \"5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf\": rpc error: code = NotFound desc = could not find container \"5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf\": container with ID starting with 5625ebaee674ebb1371b3f05d36e83333efaf1d3d729af67d7181a5baf0a90cf not found: ID does not exist" Sep 16 19:18:19.521139 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.521101 2570 scope.go:117] "RemoveContainer" containerID="b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373" Sep 16 19:18:19.521415 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:18:19.521392 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373\": container with ID starting with b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373 not found: ID does not exist" containerID="b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373" Sep 16 19:18:19.521479 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.521422 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373"} err="failed to get container status \"b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373\": rpc error: code = NotFound desc = could not find container \"b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373\": container with ID starting with b42511c9f9188685d0ad78b6ea094a96455e499f8c33c4e7d86da4c8e33de373 not found: ID does not exist" Sep 16 19:18:19.526459 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.526429 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9"] Sep 16 19:18:19.530929 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:19.530906 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-27911-predictor-6965d7c55b-x57m9"] Sep 16 19:18:20.971420 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:20.971379 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" path="/var/lib/kubelet/pods/d8777970-6d1f-4eac-b74e-8a740e4e0aa4/volumes" Sep 16 19:18:23.501262 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:23.501233 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:18:23.501770 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:23.501739 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:18:33.502319 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:33.502284 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:18:43.502135 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:43.502099 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:18:53.502603 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:18:53.502558 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused" Sep 16 19:19:03.502829 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:03.502799 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:19:15.156655 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.156626 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8"] Sep 16 19:19:15.157110 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.156849 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" containerID="cri-o://4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0" gracePeriod=30 Sep 16 19:19:15.157110 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.156917 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kube-rbac-proxy" containerID="cri-o://5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63" gracePeriod=30 Sep 16 19:19:15.358604 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.358564 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl"] Sep 16 19:19:15.359018 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.359005 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" Sep 16 19:19:15.359069 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.359020 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" Sep 16 19:19:15.359069 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.359041 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kube-rbac-proxy" Sep 16 19:19:15.359069 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.359046 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kube-rbac-proxy" Sep 16 19:19:15.359163 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.359112 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kserve-container" Sep 16 19:19:15.359163 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.359121 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8777970-6d1f-4eac-b74e-8a740e4e0aa4" containerName="kube-rbac-proxy" Sep 16 19:19:15.362186 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.362170 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.364396 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.364375 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7f5a4-predictor-serving-cert\"" Sep 16 19:19:15.364396 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.364385 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-7f5a4-kube-rbac-proxy-sar-config\"" Sep 16 19:19:15.373533 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.373503 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl"] Sep 16 19:19:15.431709 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.431635 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4pv\" (UniqueName: \"kubernetes.io/projected/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-api-access-9d4pv\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.431856 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.431748 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.431856 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.431772 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.533062 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.533007 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.533062 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.533060 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.533302 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.533107 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4pv\" (UniqueName: \"kubernetes.io/projected/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-api-access-9d4pv\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.533302 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:19:15.533162 2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-serving-cert: secret "error-404-isvc-7f5a4-predictor-serving-cert" not found Sep 16 19:19:15.533302 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:19:15.533237 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls podName:96d7a6a0-de0d-411f-8460-08e50a117de7 nodeName:}" failed. No retries permitted until 2026-09-16 19:19:16.033220484 +0000 UTC m=+2043.576526240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls") pod "error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" (UID: "96d7a6a0-de0d-411f-8460-08e50a117de7") : secret "error-404-isvc-7f5a4-predictor-serving-cert" not found Sep 16 19:19:15.533805 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.533786 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.541818 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.541771 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4pv\" (UniqueName: \"kubernetes.io/projected/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-api-access-9d4pv\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:15.785520 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.785486 2570 generic.go:356] "Generic (PLEG): container finished" podID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerID="5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63" exitCode=2 Sep 16 19:19:15.785748 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:15.785570 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" event={"ID":"a24533bc-9ebc-4233-8a25-a5d1f42c77de","Type":"ContainerDied","Data":"5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63"} Sep 16 19:19:16.037319 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.037234 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:16.039942 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.039918 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls\") pod \"error-404-isvc-7f5a4-predictor-f9d645766-gsvbl\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:16.273425 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.273386 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:16.550287 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.550212 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl"] Sep 16 19:19:16.553607 ip-10-0-142-175 kubenswrapper[2570]: W0916 19:19:16.553578 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d7a6a0_de0d_411f_8460_08e50a117de7.slice/crio-a8562220b07982a01931f3ac6ec8ddcb1adfdee32887d4aafe0ed7ec2d21ab25 WatchSource:0}: Error finding container a8562220b07982a01931f3ac6ec8ddcb1adfdee32887d4aafe0ed7ec2d21ab25: Status 404 returned error can't find the container with id a8562220b07982a01931f3ac6ec8ddcb1adfdee32887d4aafe0ed7ec2d21ab25 Sep 16 19:19:16.793260 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.793225 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" event={"ID":"96d7a6a0-de0d-411f-8460-08e50a117de7","Type":"ContainerStarted","Data":"1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461"} Sep 16 19:19:16.793439 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.793292 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" event={"ID":"96d7a6a0-de0d-411f-8460-08e50a117de7","Type":"ContainerStarted","Data":"e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9"} Sep 16 19:19:16.793439 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.793319 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" event={"ID":"96d7a6a0-de0d-411f-8460-08e50a117de7","Type":"ContainerStarted","Data":"a8562220b07982a01931f3ac6ec8ddcb1adfdee32887d4aafe0ed7ec2d21ab25"} Sep 16 19:19:16.793439 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.793345 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:16.813273 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:16.813167 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podStartSLOduration=1.813149723 podStartE2EDuration="1.813149723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:16.811436649 +0000 UTC m=+2044.354742432" watchObservedRunningTime="2026-09-16 19:19:16.813149723 +0000 UTC m=+2044.356455481" Sep 16 19:19:17.798022 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:17.797991 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:17.799773 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:17.799734 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 19:19:18.070420 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.070323 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused" Sep 16 19:19:18.074737 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.074708 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused" Sep 16 19:19:18.632223 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.632144 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:19:18.662394 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.662361 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24533bc-9ebc-4233-8a25-a5d1f42c77de-proxy-tls\") pod \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " Sep 16 19:19:18.662572 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.662479 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-rbac-proxy-sar-config\") pod \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " Sep 16 19:19:18.662572 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.662534 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmzqr\" (UniqueName: \"kubernetes.io/projected/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-api-access-gmzqr\") pod \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\" (UID: \"a24533bc-9ebc-4233-8a25-a5d1f42c77de\") " Sep 16 19:19:18.662844 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.662818 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a24533bc-9ebc-4233-8a25-a5d1f42c77de" (UID: "a24533bc-9ebc-4233-8a25-a5d1f42c77de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:19:18.664767 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.664742 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a24533bc-9ebc-4233-8a25-a5d1f42c77de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a24533bc-9ebc-4233-8a25-a5d1f42c77de" (UID: "a24533bc-9ebc-4233-8a25-a5d1f42c77de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:19:18.664936 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.664910 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-api-access-gmzqr" (OuterVolumeSpecName: "kube-api-access-gmzqr") pod "a24533bc-9ebc-4233-8a25-a5d1f42c77de" (UID: "a24533bc-9ebc-4233-8a25-a5d1f42c77de"). InnerVolumeSpecName "kube-api-access-gmzqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:19:18.764318 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.764262 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmzqr\" (UniqueName: \"kubernetes.io/projected/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-api-access-gmzqr\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:19:18.764318 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.764309 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a24533bc-9ebc-4233-8a25-a5d1f42c77de-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:19:18.764318 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.764319 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a24533bc-9ebc-4233-8a25-a5d1f42c77de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:19:18.804193 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.804159 2570 generic.go:356] "Generic (PLEG): container finished" podID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerID="4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0" exitCode=0 Sep 16 19:19:18.804588 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.804206 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" Sep 16 19:19:18.804588 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.804239 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" event={"ID":"a24533bc-9ebc-4233-8a25-a5d1f42c77de","Type":"ContainerDied","Data":"4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0"} Sep 16 19:19:18.804588 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.804307 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8" event={"ID":"a24533bc-9ebc-4233-8a25-a5d1f42c77de","Type":"ContainerDied","Data":"5fb3373102f4b34fa70f65a11f4ec36217f954287cba12a8325ac3b4ee39db12"} Sep 16 19:19:18.804588 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.804335 2570 scope.go:117] "RemoveContainer" containerID="5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63" Sep 16 19:19:18.804942 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.804921 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 19:19:18.814105 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.814086 2570 scope.go:117] "RemoveContainer" containerID="4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0" Sep 16 19:19:18.822694 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.822673 2570 scope.go:117] "RemoveContainer" containerID="5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63" Sep 16 19:19:18.823037 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:19:18.823017 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63\": container with ID starting with 5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63 not found: ID does not exist" containerID="5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63" Sep 16 19:19:18.823097 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.823052 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63"} err="failed to get container status \"5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63\": rpc error: code = NotFound desc = could not find container \"5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63\": container with ID starting with 5a59d8c416f4aabeaac2a26cbecc4618055927c3c72bb03e9a8e7a634b6efe63 not found: ID does not exist" Sep 16 19:19:18.823097 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.823074 2570 scope.go:117] "RemoveContainer" containerID="4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0" Sep 16 19:19:18.823326 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:19:18.823310 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0\": container with ID starting with 4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0 not found: ID does not exist" containerID="4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0" Sep 16 19:19:18.823375 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.823332 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0"} err="failed to get container status \"4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0\": rpc error: code = NotFound desc = could not find container \"4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0\": container with ID starting with 4ba78e4f39575e170cd19426e384dd0e08ced6b207a1e7de1c718be51c1419d0 not found: ID does not exist" Sep 16 19:19:18.826893 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.826847 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8"] Sep 16 19:19:18.833480 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.833444 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-53a10-predictor-765f8bbccb-v7px8"] Sep 16 19:19:18.971589 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:18.971562 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" path="/var/lib/kubelet/pods/a24533bc-9ebc-4233-8a25-a5d1f42c77de/volumes" Sep 16 19:19:23.810259 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:23.810229 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:19:23.810916 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:23.810855 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 19:19:25.630336 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.630299 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g"] Sep 16 19:19:25.630825 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.630540 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" containerID="cri-o://6280326b8bd4c04374b6321de22a7214db850c038467c4f9d941f44f986a7d94" gracePeriod=30 Sep 16 19:19:25.630825 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.630620 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kube-rbac-proxy" containerID="cri-o://b90c4c23dff22e50836c20a7d5400b48e5df370ba7ad50d8a6311d482aea5568" gracePeriod=30 Sep 16 19:19:25.817946 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.817909 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4"] Sep 16 19:19:25.818313 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.818300 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" Sep 16 19:19:25.818363 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.818315 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" Sep 16 19:19:25.818363 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.818339 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kube-rbac-proxy" Sep 16 19:19:25.818363 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.818344 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kube-rbac-proxy" Sep 16 19:19:25.818457 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.818399 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kube-rbac-proxy" Sep 16 19:19:25.818457 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.818409 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="a24533bc-9ebc-4233-8a25-a5d1f42c77de" containerName="kserve-container" Sep 16 19:19:25.821457 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.821442 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:25.823450 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.823420 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-17a4d-kube-rbac-proxy-sar-config\"" Sep 16 19:19:25.823580 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.823516 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-17a4d-predictor-serving-cert\"" Sep 16 19:19:25.831887 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.831839 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4"] Sep 16 19:19:25.842209 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.842175 2570 generic.go:356] "Generic (PLEG): container finished" podID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerID="b90c4c23dff22e50836c20a7d5400b48e5df370ba7ad50d8a6311d482aea5568" exitCode=2 Sep 16 19:19:25.842366 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.842251 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" event={"ID":"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1","Type":"ContainerDied","Data":"b90c4c23dff22e50836c20a7d5400b48e5df370ba7ad50d8a6311d482aea5568"} Sep 16 19:19:25.930309 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.930217 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:25.930309 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.930293 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:25.930517 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:25.930330 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpct9\" (UniqueName: \"kubernetes.io/projected/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-api-access-jpct9\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:26.031727 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:26.031685 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:26.031980 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:26.031758 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:26.031980 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:26.031793 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpct9\" (UniqueName: \"kubernetes.io/projected/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-api-access-jpct9\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:26.031980 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:19:26.031969 2570 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-serving-cert: secret "error-404-isvc-17a4d-predictor-serving-cert" not found Sep 16 19:19:26.032178 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:19:26.032041 2570 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls podName:43badc11-3b5e-4dd5-92b5-1cde83fe1b81 nodeName:}" failed. No retries permitted until 2026-09-16 19:19:26.532019366 +0000 UTC m=+2054.075325120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls") pod "error-404-isvc-17a4d-predictor-6db794b454-pbjn4" (UID: "43badc11-3b5e-4dd5-92b5-1cde83fe1b81") : secret "error-404-isvc-17a4d-predictor-serving-cert" not found Sep 16 19:19:26.032447 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:26.032426 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:26.043067 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:26.043045 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpct9\" (UniqueName: \"kubernetes.io/projected/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-api-access-jpct9\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:26.537262 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:26.537221 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:26.540023 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:26.539997 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls\") pod \"error-404-isvc-17a4d-predictor-6db794b454-pbjn4\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:26.732932 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:26.732886 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:27.015408 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:27.015365 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4"] Sep 16 19:19:27.022915 ip-10-0-142-175 kubenswrapper[2570]: W0916 19:19:27.022881 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43badc11_3b5e_4dd5_92b5_1cde83fe1b81.slice/crio-279ea29bb4de042a743d54301404d7fbfeefe5122e448813a9f72e02773aace3 WatchSource:0}: Error finding container 279ea29bb4de042a743d54301404d7fbfeefe5122e448813a9f72e02773aace3: Status 404 returned error can't find the container with id 279ea29bb4de042a743d54301404d7fbfeefe5122e448813a9f72e02773aace3 Sep 16 19:19:27.853565 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:27.853525 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" event={"ID":"43badc11-3b5e-4dd5-92b5-1cde83fe1b81","Type":"ContainerStarted","Data":"02e5c3a3b797d910234e91d80118882d7a6d9eebeee3bd349ef60b4ad079e9bb"} Sep 16 19:19:27.853970 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:27.853587 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" event={"ID":"43badc11-3b5e-4dd5-92b5-1cde83fe1b81","Type":"ContainerStarted","Data":"a4b195b6ad34a64595f73709f77d36e31193a59a04579b8f87547289813378a9"} Sep 16 19:19:27.853970 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:27.853608 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" event={"ID":"43badc11-3b5e-4dd5-92b5-1cde83fe1b81","Type":"ContainerStarted","Data":"279ea29bb4de042a743d54301404d7fbfeefe5122e448813a9f72e02773aace3"} Sep 16 19:19:27.853970 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:27.853651 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:27.871941 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:27.871893 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podStartSLOduration=2.87185624 podStartE2EDuration="2.87185624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:27.87073797 +0000 UTC m=+2055.414043728" watchObservedRunningTime="2026-09-16 19:19:27.87185624 +0000 UTC m=+2055.415161998" Sep 16 19:19:28.497390 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:28.497339 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused" Sep 16 19:19:28.860697 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:28.860664 2570 generic.go:356] "Generic (PLEG): container finished" podID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerID="6280326b8bd4c04374b6321de22a7214db850c038467c4f9d941f44f986a7d94" exitCode=0 Sep 16 19:19:28.861146 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:28.860735 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" event={"ID":"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1","Type":"ContainerDied","Data":"6280326b8bd4c04374b6321de22a7214db850c038467c4f9d941f44f986a7d94"} Sep 16 19:19:28.861146 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:28.860978 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:28.862526 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:28.862501 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 19:19:28.994441 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:28.994412 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:19:29.060698 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.060601 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-api-access-ct595\") pod \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " Sep 16 19:19:29.060698 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.060681 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-rbac-proxy-sar-config\") pod \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " Sep 16 19:19:29.061077 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.060752 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-proxy-tls\") pod \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\" (UID: \"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1\") " Sep 16 19:19:29.061077 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.061061 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" (UID: "bfdbc5b8-e91e-42fe-b995-3b89ce0638b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:19:29.063068 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.063042 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-api-access-ct595" (OuterVolumeSpecName: "kube-api-access-ct595") pod "bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" (UID: "bfdbc5b8-e91e-42fe-b995-3b89ce0638b1"). InnerVolumeSpecName "kube-api-access-ct595". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:19:29.063142 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.063077 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" (UID: "bfdbc5b8-e91e-42fe-b995-3b89ce0638b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:19:29.161300 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.161270 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:19:29.161300 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.161299 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct595\" (UniqueName: \"kubernetes.io/projected/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-api-access-ct595\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:19:29.161512 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.161312 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:19:29.868337 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.868307 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" Sep 16 19:19:29.868792 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.868344 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g" event={"ID":"bfdbc5b8-e91e-42fe-b995-3b89ce0638b1","Type":"ContainerDied","Data":"3991b6fde0de85423220faa139b56084bc5c53df0e0307afc334c2791accb87a"} Sep 16 19:19:29.868792 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.868413 2570 scope.go:117] "RemoveContainer" containerID="b90c4c23dff22e50836c20a7d5400b48e5df370ba7ad50d8a6311d482aea5568" Sep 16 19:19:29.869062 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.869036 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 19:19:29.877816 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.877797 2570 scope.go:117] "RemoveContainer" containerID="6280326b8bd4c04374b6321de22a7214db850c038467c4f9d941f44f986a7d94" Sep 16 19:19:29.890468 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.890437 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g"] Sep 16 19:19:29.892241 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:29.892218 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-3bba6-predictor-67dcb48bc6-8lb5g"] Sep 16 19:19:30.971814 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:30.971782 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" path="/var/lib/kubelet/pods/bfdbc5b8-e91e-42fe-b995-3b89ce0638b1/volumes" Sep 16 19:19:33.810744 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:33.810661 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 19:19:34.873248 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:34.873214 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:19:34.873691 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:34.873664 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 19:19:43.811050 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:43.811009 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 19:19:44.874476 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:44.874428 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 19:19:53.811719 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:53.811675 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.45:8080: connect: connection refused" Sep 16 19:19:54.874353 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:19:54.874311 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 19:20:03.812058 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:20:03.812023 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:20:04.874391 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:20:04.874351 2570 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.46:8080: connect: connection refused" Sep 16 19:20:13.083525 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:20:13.083493 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:20:13.086547 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:20:13.086518 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:20:14.875006 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:20:14.874969 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:25:13.114966 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:25:13.114934 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:25:13.118555 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:25:13.118531 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:28:40.405117 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:40.405075 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4"] Sep 16 19:28:40.405736 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:40.405566 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" containerID="cri-o://a4b195b6ad34a64595f73709f77d36e31193a59a04579b8f87547289813378a9" gracePeriod=30 Sep 16 19:28:40.405736 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:40.405655 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kube-rbac-proxy" containerID="cri-o://02e5c3a3b797d910234e91d80118882d7a6d9eebeee3bd349ef60b4ad079e9bb" gracePeriod=30 Sep 16 19:28:40.587546 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:40.587512 2570 generic.go:356] "Generic (PLEG): container finished" podID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerID="02e5c3a3b797d910234e91d80118882d7a6d9eebeee3bd349ef60b4ad079e9bb" exitCode=2 Sep 16 19:28:40.587724 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:40.587593 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" event={"ID":"43badc11-3b5e-4dd5-92b5-1cde83fe1b81","Type":"ContainerDied","Data":"02e5c3a3b797d910234e91d80118882d7a6d9eebeee3bd349ef60b4ad079e9bb"} Sep 16 19:28:43.602554 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.602521 2570 generic.go:356] "Generic (PLEG): container finished" podID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerID="a4b195b6ad34a64595f73709f77d36e31193a59a04579b8f87547289813378a9" exitCode=0 Sep 16 19:28:43.602929 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.602575 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" event={"ID":"43badc11-3b5e-4dd5-92b5-1cde83fe1b81","Type":"ContainerDied","Data":"a4b195b6ad34a64595f73709f77d36e31193a59a04579b8f87547289813378a9"} Sep 16 19:28:43.656017 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.655993 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:28:43.766080 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.765990 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpct9\" (UniqueName: \"kubernetes.io/projected/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-api-access-jpct9\") pod \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " Sep 16 19:28:43.766080 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.766062 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls\") pod \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " Sep 16 19:28:43.766339 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.766098 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-rbac-proxy-sar-config\") pod \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\" (UID: \"43badc11-3b5e-4dd5-92b5-1cde83fe1b81\") " Sep 16 19:28:43.766507 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.766475 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "43badc11-3b5e-4dd5-92b5-1cde83fe1b81" (UID: "43badc11-3b5e-4dd5-92b5-1cde83fe1b81"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:28:43.768373 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.768341 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-api-access-jpct9" (OuterVolumeSpecName: "kube-api-access-jpct9") pod "43badc11-3b5e-4dd5-92b5-1cde83fe1b81" (UID: "43badc11-3b5e-4dd5-92b5-1cde83fe1b81"). InnerVolumeSpecName "kube-api-access-jpct9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:28:43.768485 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.768372 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "43badc11-3b5e-4dd5-92b5-1cde83fe1b81" (UID: "43badc11-3b5e-4dd5-92b5-1cde83fe1b81"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:28:43.867103 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.867073 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:28:43.867103 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.867098 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpct9\" (UniqueName: \"kubernetes.io/projected/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-kube-api-access-jpct9\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:28:43.867103 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:43.867109 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/43badc11-3b5e-4dd5-92b5-1cde83fe1b81-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:28:44.614453 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:44.614421 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" Sep 16 19:28:44.614961 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:44.614449 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4" event={"ID":"43badc11-3b5e-4dd5-92b5-1cde83fe1b81","Type":"ContainerDied","Data":"279ea29bb4de042a743d54301404d7fbfeefe5122e448813a9f72e02773aace3"} Sep 16 19:28:44.614961 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:44.614548 2570 scope.go:117] "RemoveContainer" containerID="02e5c3a3b797d910234e91d80118882d7a6d9eebeee3bd349ef60b4ad079e9bb" Sep 16 19:28:44.623777 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:44.623754 2570 scope.go:117] "RemoveContainer" containerID="a4b195b6ad34a64595f73709f77d36e31193a59a04579b8f87547289813378a9" Sep 16 19:28:44.635369 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:44.635344 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4"] Sep 16 19:28:44.639166 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:44.639140 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-17a4d-predictor-6db794b454-pbjn4"] Sep 16 19:28:44.970910 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:28:44.970886 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" path="/var/lib/kubelet/pods/43badc11-3b5e-4dd5-92b5-1cde83fe1b81/volumes" Sep 16 19:30:13.147387 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:30:13.147298 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:30:13.151873 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:30:13.151838 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:35:13.183401 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:35:13.183368 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:35:13.189747 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:35:13.189725 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:36:34.426580 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:34.426498 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl"] Sep 16 19:36:34.427157 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:34.426731 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" containerID="cri-o://e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9" gracePeriod=30 Sep 16 19:36:34.427157 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:34.426779 2570 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kube-rbac-proxy" containerID="cri-o://1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461" gracePeriod=30 Sep 16 19:36:34.835784 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:34.835748 2570 generic.go:356] "Generic (PLEG): container finished" podID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerID="1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461" exitCode=2 Sep 16 19:36:34.835971 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:34.835813 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" event={"ID":"96d7a6a0-de0d-411f-8460-08e50a117de7","Type":"ContainerDied","Data":"1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461"} Sep 16 19:36:37.575040 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.575010 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:36:37.695527 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.695437 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d4pv\" (UniqueName: \"kubernetes.io/projected/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-api-access-9d4pv\") pod \"96d7a6a0-de0d-411f-8460-08e50a117de7\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " Sep 16 19:36:37.695712 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.695541 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls\") pod \"96d7a6a0-de0d-411f-8460-08e50a117de7\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " Sep 16 19:36:37.695712 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.695625 2570 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-rbac-proxy-sar-config\") pod \"96d7a6a0-de0d-411f-8460-08e50a117de7\" (UID: \"96d7a6a0-de0d-411f-8460-08e50a117de7\") " Sep 16 19:36:37.696037 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.696002 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96d7a6a0-de0d-411f-8460-08e50a117de7" (UID: "96d7a6a0-de0d-411f-8460-08e50a117de7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:36:37.697795 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.697773 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96d7a6a0-de0d-411f-8460-08e50a117de7" (UID: "96d7a6a0-de0d-411f-8460-08e50a117de7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:36:37.697937 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.697893 2570 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-api-access-9d4pv" (OuterVolumeSpecName: "kube-api-access-9d4pv") pod "96d7a6a0-de0d-411f-8460-08e50a117de7" (UID: "96d7a6a0-de0d-411f-8460-08e50a117de7"). InnerVolumeSpecName "kube-api-access-9d4pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:36:37.797067 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.797022 2570 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96d7a6a0-de0d-411f-8460-08e50a117de7-proxy-tls\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:36:37.797067 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.797056 2570 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:36:37.797067 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.797066 2570 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4pv\" (UniqueName: \"kubernetes.io/projected/96d7a6a0-de0d-411f-8460-08e50a117de7-kube-api-access-9d4pv\") on node \"ip-10-0-142-175.ec2.internal\" DevicePath \"\"" Sep 16 19:36:37.852153 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.852122 2570 generic.go:356] "Generic (PLEG): container finished" podID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerID="e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9" exitCode=0 Sep 16 19:36:37.852350 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.852177 2570 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" Sep 16 19:36:37.852350 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.852199 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" event={"ID":"96d7a6a0-de0d-411f-8460-08e50a117de7","Type":"ContainerDied","Data":"e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9"} Sep 16 19:36:37.852350 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.852259 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl" event={"ID":"96d7a6a0-de0d-411f-8460-08e50a117de7","Type":"ContainerDied","Data":"a8562220b07982a01931f3ac6ec8ddcb1adfdee32887d4aafe0ed7ec2d21ab25"} Sep 16 19:36:37.852350 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.852293 2570 scope.go:117] "RemoveContainer" containerID="1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461" Sep 16 19:36:37.861601 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.861580 2570 scope.go:117] "RemoveContainer" containerID="e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9" Sep 16 19:36:37.869986 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.869962 2570 scope.go:117] "RemoveContainer" containerID="1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461" Sep 16 19:36:37.870329 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:36:37.870310 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461\": container with ID starting with 1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461 not found: ID does not exist" containerID="1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461" Sep 16 19:36:37.870387 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.870341 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461"} err="failed to get container status \"1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461\": rpc error: code = NotFound desc = could not find container \"1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461\": container with ID starting with 1fdb62f95270ba442adb732aa9fe92cfac37e099741ced2f3485d7a859789461 not found: ID does not exist" Sep 16 19:36:37.870387 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.870362 2570 scope.go:117] "RemoveContainer" containerID="e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9" Sep 16 19:36:37.870689 ip-10-0-142-175 kubenswrapper[2570]: E0916 19:36:37.870667 2570 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9\": container with ID starting with e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9 not found: ID does not exist" containerID="e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9" Sep 16 19:36:37.870774 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.870694 2570 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9"} err="failed to get container status \"e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9\": rpc error: code = NotFound desc = could not find container \"e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9\": container with ID starting with e0aba0d33d767f06094df044e7fe151e0f119a5f983cd87600306d183a2024a9 not found: ID does not exist" Sep 16 19:36:37.872952 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.872928 2570 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl"] Sep 16 19:36:37.876623 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:37.876599 2570 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-7f5a4-predictor-f9d645766-gsvbl"] Sep 16 19:36:38.971545 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:36:38.971506 2570 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" path="/var/lib/kubelet/pods/96d7a6a0-de0d-411f-8460-08e50a117de7/volumes" Sep 16 19:37:10.097231 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:10.097200 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-r4877_74b974f7-2018-46ff-8bce-901fb9018019/global-pull-secret-syncer/0.log" Sep 16 19:37:10.213349 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:10.213317 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-q78m7_29f12a00-96bd-4697-a24f-88f121e439b0/konnectivity-agent/0.log" Sep 16 19:37:10.287451 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:10.287416 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-175.ec2.internal_76eb21753bebddd59a75841e3efb8eea/haproxy/0.log" Sep 16 19:37:13.950605 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:13.950576 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c8ef80e-b4d9-4c76-b595-24e84ce6730a/alertmanager/0.log" Sep 16 19:37:13.988375 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:13.988331 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c8ef80e-b4d9-4c76-b595-24e84ce6730a/config-reloader/0.log" Sep 16 19:37:14.014945 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.014894 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c8ef80e-b4d9-4c76-b595-24e84ce6730a/kube-rbac-proxy-web/0.log" Sep 16 19:37:14.047168 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.047137 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c8ef80e-b4d9-4c76-b595-24e84ce6730a/kube-rbac-proxy/0.log" Sep 16 19:37:14.090061 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.090035 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c8ef80e-b4d9-4c76-b595-24e84ce6730a/kube-rbac-proxy-metric/0.log" Sep 16 19:37:14.122949 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.122924 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c8ef80e-b4d9-4c76-b595-24e84ce6730a/prom-label-proxy/0.log" Sep 16 19:37:14.155539 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.155518 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5c8ef80e-b4d9-4c76-b595-24e84ce6730a/init-config-reloader/0.log" Sep 16 19:37:14.337992 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.337964 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69dd66586b-btdkd_5de4aef8-79a8-4987-8358-1cc178e38454/metrics-server/0.log" Sep 16 19:37:14.379141 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.379116 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-j4v26_19beda34-68cc-4379-8bf4-7d41f797995e/monitoring-plugin/0.log" Sep 16 19:37:14.620886 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.620788 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwd8j_ee5fee63-a1d4-432a-928b-5946e3253bf4/node-exporter/0.log" Sep 16 19:37:14.646355 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.646335 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwd8j_ee5fee63-a1d4-432a-928b-5946e3253bf4/kube-rbac-proxy/0.log" Sep 16 19:37:14.677832 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.677805 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rwd8j_ee5fee63-a1d4-432a-928b-5946e3253bf4/init-textfile/0.log" Sep 16 19:37:14.718963 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.718938 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sjmf8_798f8fb8-07f0-4c07-84ff-56d9abb88cd9/kube-rbac-proxy-main/0.log" Sep 16 19:37:14.744978 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.744940 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sjmf8_798f8fb8-07f0-4c07-84ff-56d9abb88cd9/kube-rbac-proxy-self/0.log" Sep 16 19:37:14.773126 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:14.773101 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-sjmf8_798f8fb8-07f0-4c07-84ff-56d9abb88cd9/openshift-state-metrics/0.log" Sep 16 19:37:15.070218 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:15.070192 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-lqblt_ba9411be-e65f-428e-86ef-fd4284a981cf/prometheus-operator-admission-webhook/0.log" Sep 16 19:37:15.103753 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:15.103725 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d5f6d88b-t22cc_5c022665-6622-4270-b4ea-7406efdead2d/telemeter-client/0.log" Sep 16 19:37:15.127768 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:15.127742 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d5f6d88b-t22cc_5c022665-6622-4270-b4ea-7406efdead2d/reload/0.log" Sep 16 19:37:15.159429 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:15.159404 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-76d5f6d88b-t22cc_5c022665-6622-4270-b4ea-7406efdead2d/kube-rbac-proxy/0.log" Sep 16 19:37:17.632880 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:17.632844 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6844df6d4c-ndbx2_c5b17e5a-7ae2-4279-af3a-3a28080b81c9/console/0.log" Sep 16 19:37:17.668916 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:17.668854 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-qxsx2_feed169e-b4a9-476f-82a3-250cc36f122e/download-server/0.log" Sep 16 19:37:18.193471 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.193439 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-s8zb4_bf8a0586-521c-431c-97f2-6d8faf653e38/volume-data-source-validator/0.log" Sep 16 19:37:18.309391 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309353 2570 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m"] Sep 16 19:37:18.309727 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309711 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" Sep 16 19:37:18.309727 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309726 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309741 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309751 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309764 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309774 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309798 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309805 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309821 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309830 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309842 2570 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309850 2570 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309942 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309954 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kserve-container" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309963 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kserve-container" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309972 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="43badc11-3b5e-4dd5-92b5-1cde83fe1b81" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309981 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d7a6a0-de0d-411f-8460-08e50a117de7" containerName="kube-rbac-proxy" Sep 16 19:37:18.309996 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.309988 2570 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfdbc5b8-e91e-42fe-b995-3b89ce0638b1" containerName="kserve-container" Sep 16 19:37:18.314342 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.314323 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.316695 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.316670 2570 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grqt8\"/\"default-dockercfg-k5d84\"" Sep 16 19:37:18.316695 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.316692 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grqt8\"/\"kube-root-ca.crt\"" Sep 16 19:37:18.317140 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.317122 2570 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grqt8\"/\"openshift-service-ca.crt\"" Sep 16 19:37:18.321546 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.321488 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m"] Sep 16 19:37:18.354032 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.353995 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rvj7\" (UniqueName: \"kubernetes.io/projected/5b5b18ae-774d-479f-b648-d09743813d73-kube-api-access-4rvj7\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.354220 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.354040 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-sys\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.354220 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.354084 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-proc\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.354220 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.354103 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-podres\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.354220 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.354172 2570 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-lib-modules\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455237 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455144 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-proc\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455237 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455181 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-podres\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455237 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455225 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-lib-modules\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455453 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455259 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rvj7\" (UniqueName: \"kubernetes.io/projected/5b5b18ae-774d-479f-b648-d09743813d73-kube-api-access-4rvj7\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455453 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455271 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-proc\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455453 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455282 2570 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-sys\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455453 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455397 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-podres\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455453 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455417 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-sys\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.455613 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.455502 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5b5b18ae-774d-479f-b648-d09743813d73-lib-modules\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.462660 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.462636 2570 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rvj7\" (UniqueName: \"kubernetes.io/projected/5b5b18ae-774d-479f-b648-d09743813d73-kube-api-access-4rvj7\") pod \"perf-node-gather-daemonset-sxp4m\" (UID: \"5b5b18ae-774d-479f-b648-d09743813d73\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.625565 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.625524 2570 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:18.902652 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:18.902627 2570 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m"] Sep 16 19:37:18.905359 ip-10-0-142-175 kubenswrapper[2570]: W0916 19:37:18.905326 2570 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5b5b18ae_774d_479f_b648_d09743813d73.slice/crio-029d6afa4f03bd4a6564d3ca2b91f08c0bb3a385ec6f9b9cdc10208808cfe643 WatchSource:0}: Error finding container 029d6afa4f03bd4a6564d3ca2b91f08c0bb3a385ec6f9b9cdc10208808cfe643: Status 404 returned error can't find the container with id 029d6afa4f03bd4a6564d3ca2b91f08c0bb3a385ec6f9b9cdc10208808cfe643 Sep 16 19:37:19.053021 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.052978 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" event={"ID":"5b5b18ae-774d-479f-b648-d09743813d73","Type":"ContainerStarted","Data":"c0242dfb13793085cd02a9e2e50e8a06035532a9b730c5dc7f6dc3af608b006f"} Sep 16 19:37:19.053215 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.053039 2570 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" event={"ID":"5b5b18ae-774d-479f-b648-d09743813d73","Type":"ContainerStarted","Data":"029d6afa4f03bd4a6564d3ca2b91f08c0bb3a385ec6f9b9cdc10208808cfe643"} Sep 16 19:37:19.053215 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.053155 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:19.073143 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.073088 2570 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" podStartSLOduration=1.073068315 podStartE2EDuration="1.073068315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:19.070458069 +0000 UTC m=+3126.613763829" watchObservedRunningTime="2026-09-16 19:37:19.073068315 +0000 UTC m=+3126.616374075" Sep 16 19:37:19.202201 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.202116 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvb5t_fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e/dns/0.log" Sep 16 19:37:19.225314 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.225280 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-pvb5t_fc62a3fd-a7a7-45dd-bb4d-c2926051eb5e/kube-rbac-proxy/0.log" Sep 16 19:37:19.298322 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.298294 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-94bzk_3953984a-4756-4979-9d8d-1f594e70ebd3/dns-node-resolver/0.log" Sep 16 19:37:19.819176 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.819146 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cb9cc4895-rdq86_6867e786-9ad6-4582-92ef-9af500f631f1/registry/0.log" Sep 16 19:37:19.897346 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:19.897310 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q26jb_bdd72ad2-4fcc-48c2-8bce-1790994113ad/node-ca/0.log" Sep 16 19:37:21.021484 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:21.021451 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-95rsc_22ca6920-60cc-4b51-8e27-f3647ad138e5/serve-healthcheck-canary/0.log" Sep 16 19:37:21.396093 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:21.396012 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-dws7b_d16ac85c-afc9-484d-8513-fa3f16202d79/insights-operator/0.log" Sep 16 19:37:21.397550 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:21.397530 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-dws7b_d16ac85c-afc9-484d-8513-fa3f16202d79/insights-operator/1.log" Sep 16 19:37:21.495547 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:21.495517 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jkdgk_541599ca-7204-4864-a897-ccc1d5d0a5ca/kube-rbac-proxy/0.log" Sep 16 19:37:21.517738 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:21.517711 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jkdgk_541599ca-7204-4864-a897-ccc1d5d0a5ca/exporter/0.log" Sep 16 19:37:21.541719 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:21.541690 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jkdgk_541599ca-7204-4864-a897-ccc1d5d0a5ca/extractor/0.log" Sep 16 19:37:23.851227 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:23.851189 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-764fc98f48-wrrs4_96012204-f109-4e7f-932c-ab5c0f39ffcc/manager/0.log" Sep 16 19:37:23.885676 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:23.885641 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-fz4mr_6cfa057a-4f53-4639-bbc2-a81daeebc6b9/manager/0.log" Sep 16 19:37:23.911008 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:23.910983 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-h4rhl_2bac9c4b-3ce7-44ea-ac1d-190d8f1ca552/server/0.log" Sep 16 19:37:25.068466 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:25.068440 2570 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-sxp4m" Sep 16 19:37:29.719915 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:29.719882 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-78bv4_dcf8d8d6-7154-4f6e-95ef-b7894c9d5ab5/kube-multus/0.log" Sep 16 19:37:29.743427 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:29.743397 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wszm_9a925473-9403-4198-a8cf-2ac41b9e73d0/kube-multus-additional-cni-plugins/0.log" Sep 16 19:37:29.765686 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:29.765660 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wszm_9a925473-9403-4198-a8cf-2ac41b9e73d0/egress-router-binary-copy/0.log" Sep 16 19:37:29.788459 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:29.788432 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wszm_9a925473-9403-4198-a8cf-2ac41b9e73d0/cni-plugins/0.log" Sep 16 19:37:29.811838 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:29.811812 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wszm_9a925473-9403-4198-a8cf-2ac41b9e73d0/bond-cni-plugin/0.log" Sep 16 19:37:29.833501 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:29.833470 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wszm_9a925473-9403-4198-a8cf-2ac41b9e73d0/routeoverride-cni/0.log" Sep 16 19:37:29.858214 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:29.858187 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wszm_9a925473-9403-4198-a8cf-2ac41b9e73d0/whereabouts-cni-bincopy/0.log" Sep 16 19:37:29.899353 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:29.899327 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4wszm_9a925473-9403-4198-a8cf-2ac41b9e73d0/whereabouts-cni/0.log" Sep 16 19:37:30.426180 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:30.426150 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mllt4_f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba/network-metrics-daemon/0.log" Sep 16 19:37:30.449008 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:30.448979 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-mllt4_f7f7e4b9-8c62-41f6-a0c9-71f149faa2ba/kube-rbac-proxy/0.log" Sep 16 19:37:31.425300 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.425272 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-controller/0.log" Sep 16 19:37:31.444819 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.444791 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/0.log" Sep 16 19:37:31.445129 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.445116 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovn-acl-logging/1.log" Sep 16 19:37:31.465290 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.465263 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/kube-rbac-proxy-node/0.log" Sep 16 19:37:31.488261 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.488226 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 19:37:31.508016 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.507989 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/northd/0.log" Sep 16 19:37:31.528482 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.528453 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/nbdb/0.log" Sep 16 19:37:31.554621 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.554596 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/sbdb/0.log" Sep 16 19:37:31.597051 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:31.597017 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-nz9xs_21641444-a6be-4e57-8f10-7bf854e3ab77/ovnkube-controller/0.log" Sep 16 19:37:32.769881 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:32.769830 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-pprg4_26fa1a05-58b1-4135-9703-6e7775f03691/check-endpoints/0.log" Sep 16 19:37:32.843425 ip-10-0-142-175 kubenswrapper[2570]: I0916 19:37:32.843390 2570 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-pxxb4_c2232d20-f65c-48c0-8165-64844c147398/network-check-target-container/0.log"