Jul 09 17:39:30.583718 ip-10-0-141-169 systemd[1]: Starting Kubernetes Kubelet... Jul 09 17:39:32.598494 ip-10-0-141-169 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 17:39:32.598494 ip-10-0-141-169 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 17:39:32.598494 ip-10-0-141-169 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 17:39:32.598494 ip-10-0-141-169 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 17:39:32.598494 ip-10-0-141-169 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 17:39:32.599234 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.598558 2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 17:39:32.602202 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602179 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 17:39:32.602202 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602200 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 17:39:32.602202 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602204 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602208 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602211 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602214 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602216 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602220 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602223 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602226 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602229 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602232 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602235 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602237 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602240 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602243 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602247 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602250 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602253 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602255 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602258 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602261 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 17:39:32.602304 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602264 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602266 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602269 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602272 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602274 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602277 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602279 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602282 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602284 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602287 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602290 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602293 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602295 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602298 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602303 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602306 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602309 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602312 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602315 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602318 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 17:39:32.602798 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602321 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602323 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602326 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602328 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602331 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602334 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602336 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602340 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602342 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602344 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602347 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602350 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602352 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602355 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602358 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602360 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602363 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602365 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602368 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602370 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 17:39:32.603278 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602373 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602375 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602378 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602381 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602383 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602386 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602392 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602396 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602399 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602408 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602412 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602428 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602431 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602434 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602438 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602440 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602443 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602446 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602448 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 17:39:32.603774 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602451 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602454 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602456 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602459 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602461 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602464 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602468 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602471 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602473 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602476 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602479 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602483 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602486 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602490 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602493 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602496 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602499 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.602502 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603235 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 17:39:32.604237 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603242 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603246 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603250 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603253 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603256 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603259 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603262 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603265 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603268 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603271 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603273 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603276 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603279 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603282 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603285 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603287 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603290 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603294 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603296 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 17:39:32.604711 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603299 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603301 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603304 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603307 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603309 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603312 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603314 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603317 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603319 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603322 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603325 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603327 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603330 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603334 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603337 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603340 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603344 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603348 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603352 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 17:39:32.605176 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603355 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603358 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603361 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603366 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603369 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603372 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603375 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603379 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603381 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603384 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603387 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603389 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603392 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603395 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603397 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603400 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603402 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603405 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603407 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 17:39:32.605659 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603410 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603412 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603434 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603439 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603442 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603446 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603450 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603458 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603461 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603463 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603466 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603468 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603471 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603474 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603477 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603480 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603483 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603486 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603488 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603491 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 17:39:32.606124 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603494 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603499 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603502 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603505 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603508 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603510 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603516 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603519 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603521 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603524 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603527 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603530 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603532 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603535 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603538 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603541 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603544 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603546 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603549 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603553 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 17:39:32.606643 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.603555 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604005 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604011 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604014 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604018 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604021 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604026 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604029 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604032 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604034 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604037 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604039 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604042 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604046 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604049 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604052 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604055 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604057 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604062 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604065 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 17:39:32.607129 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604068 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604070 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604073 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604076 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604079 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604081 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604084 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604086 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604089 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604091 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604094 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604098 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604100 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604103 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604105 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604108 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604110 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604113 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604116 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604119 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 17:39:32.607646 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604121 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604124 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604126 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604129 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604131 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604135 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604137 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604140 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604142 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604145 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604149 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604152 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604154 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604157 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604159 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604162 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604165 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604167 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604170 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604172 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 17:39:32.608144 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604175 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604177 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604180 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604183 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604186 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604189 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604191 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604194 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604197 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604199 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604202 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604205 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604208 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604210 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604213 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604215 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604217 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604220 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604222 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604225 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 17:39:32.608636 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604227 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604230 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604234 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604237 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604240 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604244 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604248 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604251 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604255 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604259 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604263 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604265 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604268 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604271 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604274 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604277 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604280 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604282 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604285 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 17:39:32.609161 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.604289 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604368 2560 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604379 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604401 2560 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604433 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604440 2560 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604444 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604448 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604453 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604456 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604459 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604463 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604467 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604471 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604474 2560 flags.go:64] FLAG: --cgroup-root="" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604479 2560 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604482 2560 flags.go:64] FLAG: --client-ca-file="" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604485 2560 flags.go:64] FLAG: --cloud-provider="external" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604488 2560 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604493 2560 flags.go:64] FLAG: --cluster-domain="" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604496 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604499 2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604503 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604506 2560 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 17:39:32.609643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604510 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604513 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604516 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604520 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604524 2560 flags.go:64] FLAG: --contention-profiling="false" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604527 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604530 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604534 2560 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604537 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604541 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604544 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604547 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604551 2560 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604554 2560 flags.go:64] FLAG: --enable-server="true" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604557 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604562 2560 flags.go:64] FLAG: --event-burst="100" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604566 2560 flags.go:64] FLAG: --event-qps="50" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604569 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604572 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604575 2560 flags.go:64] FLAG: --eviction-hard="" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604579 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604582 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604585 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604590 2560 flags.go:64] FLAG: --eviction-soft="" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604593 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 17:39:32.610213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604596 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604599 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604603 2560 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604606 2560 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604609 2560 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604612 2560 flags.go:64] FLAG: --feature-gates="" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604616 2560 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604619 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604622 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604626 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604629 2560 flags.go:64] FLAG: --healthz-port="10248" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604634 2560 flags.go:64] FLAG: --help="false" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604637 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604640 2560 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604643 2560 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604646 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604650 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604654 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604657 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604660 2560 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604663 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604666 2560 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604669 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604673 2560 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 17:39:32.610878 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604676 2560 flags.go:64] FLAG: --kube-reserved="" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604679 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604682 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604685 2560 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604688 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604691 2560 flags.go:64] FLAG: --lock-file="" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604696 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604699 2560 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604702 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604708 2560 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604711 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604715 2560 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604718 2560 flags.go:64] FLAG: --logging-format="text" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604721 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604725 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604728 2560 flags.go:64] FLAG: --manifest-url="" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604731 2560 flags.go:64] FLAG: --manifest-url-header="" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604736 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604739 2560 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604745 2560 flags.go:64] FLAG: --max-pods="110" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604748 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604752 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604755 2560 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604758 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604761 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 17:39:32.611481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604764 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604767 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604775 2560 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604778 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604781 2560 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604784 2560 flags.go:64] FLAG: --pod-cidr="" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604788 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604795 2560 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604798 2560 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604801 2560 flags.go:64] FLAG: --pods-per-core="0" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604804 2560 flags.go:64] FLAG: --port="10250" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604807 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604811 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09d9e7d0b50622318" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604814 2560 flags.go:64] FLAG: --qos-reserved="" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604819 2560 flags.go:64] FLAG: --read-only-port="10255" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604822 2560 flags.go:64] FLAG: --register-node="true" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604825 2560 flags.go:64] FLAG: --register-with-taints="" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604829 2560 flags.go:64] FLAG: --registry-burst="10" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604832 2560 flags.go:64] FLAG: --registry-qps="5" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604835 2560 flags.go:64] FLAG: --reserved-cpus="" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604838 2560 flags.go:64] FLAG: --reserved-memory="" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604842 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604846 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604849 2560 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604852 2560 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 17:39:32.612105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604855 2560 flags.go:64] FLAG: --runonce="false" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604860 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604863 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604867 2560 flags.go:64] FLAG: --seccomp-default="false" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604870 2560 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604873 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604876 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604879 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604882 2560 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604885 2560 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604889 2560 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604891 2560 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604894 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604898 2560 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604901 2560 flags.go:64] FLAG: --system-cgroups="" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604904 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604910 2560 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604913 2560 flags.go:64] FLAG: --tls-cert-file="" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604916 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604920 2560 flags.go:64] FLAG: --tls-min-version="" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604923 2560 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604928 2560 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604931 2560 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604934 2560 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604937 2560 flags.go:64] FLAG: --v="2" Jul 09 17:39:32.612736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604941 2560 flags.go:64] FLAG: --version="false" Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604945 2560 flags.go:64] FLAG: --vmodule="" Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604950 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.604953 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605076 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605081 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605084 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605087 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605092 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605095 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605097 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605100 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605103 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605105 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605108 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605110 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605113 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605116 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605118 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605121 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605123 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 17:39:32.613454 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605126 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605130 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605133 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605136 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605138 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605141 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605143 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605149 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605152 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605154 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605157 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605160 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605162 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605165 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605168 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605170 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605173 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605175 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605178 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605184 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 17:39:32.613968 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605187 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605190 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605193 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605195 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605198 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605201 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605203 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605206 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605208 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605211 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605213 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605216 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605218 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605221 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605223 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605226 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605229 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605231 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605234 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605239 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 17:39:32.614496 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605241 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605244 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605247 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605249 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605252 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605254 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605257 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605259 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605262 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605264 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605266 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605270 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605273 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605277 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605281 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605284 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605287 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605290 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605293 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 17:39:32.615003 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605296 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605299 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605302 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605304 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605307 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605310 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605313 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605315 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605318 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605321 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605323 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605326 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605330 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605334 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605337 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605340 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605342 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605345 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605347 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 17:39:32.615493 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605350 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605352 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605355 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.605357 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.605976 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.613126 2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.613145 2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.613163 2560 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.613169 2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613231 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613237 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613240 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613243 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613247 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 17:39:32.616206 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613250 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613253 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613256 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613259 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613262 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613264 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613267 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613269 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613272 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613275 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613278 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613280 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613283 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613286 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613288 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613292 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613294 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613297 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613300 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613302 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613305 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 17:39:32.616730 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613307 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613310 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613313 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613317 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613320 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613323 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613325 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613328 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613331 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613333 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613337 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613341 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613343 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613347 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613350 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613353 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613356 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613358 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613361 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 17:39:32.617457 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613363 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613368 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613371 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613374 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613378 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613381 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613384 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613388 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613392 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613395 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613398 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613400 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613403 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613406 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613408 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613412 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613429 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613433 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613436 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 17:39:32.617926 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613438 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613441 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613444 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613447 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613449 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613452 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613455 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613458 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613460 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613463 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613466 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613469 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613472 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613474 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613477 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613480 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613483 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613487 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613489 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613492 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 17:39:32.618403 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613494 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613497 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613500 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613502 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613505 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613509 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613513 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613516 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613518 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613521 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613524 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613527 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613530 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613532 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 17:39:32.619034 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613535 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.613540 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613662 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613667 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613671 2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613674 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613677 2560 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613680 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613682 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613685 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613688 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613690 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613693 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613697 2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613701 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 17:39:32.619384 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613704 2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613707 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613710 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613713 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613715 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613718 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613720 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613723 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613726 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613728 2560 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613731 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613734 2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613736 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613739 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613741 2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613746 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613749 2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613752 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613755 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 17:39:32.619783 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613758 2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613761 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613764 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613766 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613769 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613772 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613774 2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613777 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613779 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613782 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613785 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613788 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613790 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613793 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613795 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613798 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613801 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613803 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613806 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613808 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 17:39:32.620266 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613811 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613813 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613816 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613818 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613822 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613824 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613827 2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613830 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613832 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613836 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613838 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613841 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613843 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613846 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613849 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613852 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613854 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613857 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613860 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 17:39:32.620788 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613864 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613867 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613869 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613871 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613874 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613876 2560 feature_gate.go:328] unrecognized feature gate: Example Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613879 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613882 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613885 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613887 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613890 2560 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613893 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613895 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613898 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613901 2560 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613903 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613906 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613909 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613911 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613914 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 17:39:32.621258 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613916 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613919 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613921 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613924 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613927 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613929 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613932 2560 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:32.613935 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.613941 2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.614098 2560 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.618956 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.620235 2560 server.go:1019] "Starting client certificate rotation" Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.620341 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 17:39:32.621766 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.620378 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 17:39:32.622863 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.622844 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 17:39:32.624167 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.624152 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 17:39:32.634527 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.634499 2560 log.go:25] "Validated CRI v1 runtime API" Jul 09 17:39:32.636758 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.636737 2560 log.go:25] "Validated CRI v1 image API" Jul 09 17:39:32.639906 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.639884 2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 17:39:32.642466 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.642439 2560 fs.go:135] Filesystem UUIDs: map[2d84da77-a77e-4142-a2ee-d244630b300d:/dev/nvme0n1p3 6f50824b-d274-4775-a6aa-75eb0f14db16:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 09 17:39:32.642579 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.642464 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 17:39:32.647738 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.647587 2560 manager.go:216] Machine: {Timestamp:2026-07-09 17:39:32.646402673 +0000 UTC m=+0.348903854 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101947 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec240bc42f46d769e8ec899fb40eced6 SystemUUID:ec240bc4-2f46-d769-e8ec-899fb40eced6 BootID:bd06de47-d7b2-4d4b-ac72-3a73f7e9bd49 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c6:38:e3:7c:61 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c6:38:e3:7c:61 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5a:19:70:4c:25:a4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 17:39:32.647738 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.647716 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 17:39:32.647967 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.647874 2560 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 17:39:32.650222 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.650200 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 17:39:32.650768 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.650738 2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 17:39:32.650929 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.650773 2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-169.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 17:39:32.651482 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.651470 2560 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 17:39:32.651515 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.651485 2560 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 17:39:32.651515 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.651494 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 17:39:32.652358 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.652344 2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 17:39:32.653312 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.653297 2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 17:39:32.656615 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.656599 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 09 17:39:32.656759 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.656749 2560 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 17:39:32.656800 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.656782 2560 kubelet.go:486] "Attempting to sync node with API server" Jul 09 17:39:32.656800 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.656792 2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 17:39:32.656868 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.656806 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 17:39:32.656868 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.656818 2560 kubelet.go:398] "Adding apiserver pod source" Jul 09 17:39:32.656868 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.656829 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 17:39:32.657298 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.657285 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 17:39:32.657364 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.657305 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 17:39:32.661020 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.660953 2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 09 17:39:32.662921 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.662899 2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 17:39:32.663030 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.662940 2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 17:39:32.663658 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663642 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 17:39:32.663726 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663664 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 17:39:32.663726 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663674 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 17:39:32.663726 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663682 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 17:39:32.663726 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663695 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 17:39:32.663726 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663703 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 17:39:32.663726 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663716 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 17:39:32.663726 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663723 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 17:39:32.664044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663735 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 17:39:32.664044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663744 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 17:39:32.664044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663756 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 17:39:32.664044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663764 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 17:39:32.664044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663795 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 17:39:32.664044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.663812 2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 17:39:32.667269 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.667250 2560 server.go:1268] "Started kubelet" Jul 09 17:39:32.667401 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.667343 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 17:39:32.667521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.667334 2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 17:39:32.667588 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.667550 2560 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 17:39:32.668208 ip-10-0-141-169 systemd[1]: Started Kubernetes Kubelet. Jul 09 17:39:32.668321 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.668264 2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-169.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 17:39:32.668501 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.668467 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 17:39:32.668603 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.668583 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-169.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 17:39:32.670505 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.670486 2560 server.go:310] "Adding debug handlers to kubelet server" Jul 09 17:39:32.672567 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.672548 2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 17:39:32.673741 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.673708 2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 17:39:32.676058 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.675198 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-169.ec2.internal.18c0afd2df8c4b46 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-169.ec2.internal,UID:ip-10-0-141-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-169.ec2.internal,},FirstTimestamp:2026-07-09 17:39:32.667222854 +0000 UTC m=+0.369724035,LastTimestamp:2026-07-09 17:39:32.667222854 +0000 UTC m=+0.369724035,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-169.ec2.internal,}" Jul 09 17:39:32.677145 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.677128 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 17:39:32.677215 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.677142 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 17:39:32.677734 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.677310 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:32.677734 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.677624 2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 17:39:32.677734 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.677624 2560 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 17:39:32.677734 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.677662 2560 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 17:39:32.677946 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.677804 2560 reconstruct.go:97] "Volume reconstruction finished" Jul 09 17:39:32.677946 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.677812 2560 reconciler.go:29] "Reconciler: start to sync state" Jul 09 17:39:32.678084 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.678066 2560 factory.go:55] Registering systemd factory Jul 09 17:39:32.678157 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.678089 2560 factory.go:223] Registration of the systemd container factory successfully Jul 09 17:39:32.678374 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.678360 2560 factory.go:153] Registering CRI-O factory Jul 09 17:39:32.678451 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.678377 2560 factory.go:223] Registration of the crio container factory successfully Jul 09 17:39:32.678451 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.678441 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 17:39:32.678542 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.678470 2560 factory.go:103] Registering Raw factory Jul 09 17:39:32.678542 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.678491 2560 manager.go:1196] Started watching for new ooms in manager Jul 09 17:39:32.679066 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.679044 2560 manager.go:349] Starting recovery of all containers Jul 09 17:39:32.682133 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.682102 2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 17:39:32.682275 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.682247 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-169.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 17:39:32.692114 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.691528 2560 manager.go:354] Recovery completed Jul 09 17:39:32.696745 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.696727 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 17:39:32.700537 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.700508 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientMemory" Jul 09 17:39:32.700615 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.700552 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 17:39:32.700615 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.700566 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientPID" Jul 09 17:39:32.701207 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.701190 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 17:39:32.701207 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.701200 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 17:39:32.701305 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.701206 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 17:39:32.701305 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.701256 2560 state_mem.go:36] "Initialized new in-memory state store" Jul 09 17:39:32.701976 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.701958 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientMemory" Jul 09 17:39:32.702025 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.701998 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 17:39:32.702025 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.702017 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientPID" Jul 09 17:39:32.702382 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.702290 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-169.ec2.internal.18c0afd2e1889e5b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-169.ec2.internal,UID:ip-10-0-141-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-169.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-169.ec2.internal,},FirstTimestamp:2026-07-09 17:39:32.700536411 +0000 UTC m=+0.403037596,LastTimestamp:2026-07-09 17:39:32.700536411 +0000 UTC m=+0.403037596,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-169.ec2.internal,}" Jul 09 17:39:32.702591 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.702575 2560 policy_none.go:49] "None policy: Start" Jul 09 17:39:32.702679 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.702596 2560 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 17:39:32.702679 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.702611 2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 17:39:32.703628 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.703616 2560 policy_none.go:47] "Start" Jul 09 17:39:32.707406 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.707385 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wxskc" Jul 09 17:39:32.715944 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.715859 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-169.ec2.internal.18c0afd2e188f81a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-169.ec2.internal,UID:ip-10-0-141-169.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-169.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-169.ec2.internal,},FirstTimestamp:2026-07-09 17:39:32.700559386 +0000 UTC m=+0.403060567,LastTimestamp:2026-07-09 17:39:32.700559386 +0000 UTC m=+0.403060567,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-169.ec2.internal,}" Jul 09 17:39:32.718943 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.718912 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wxskc" Jul 09 17:39:32.748203 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.748179 2560 manager.go:337] "Starting Device Plugin manager" Jul 09 17:39:32.748388 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.748232 2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 17:39:32.748388 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.748246 2560 server.go:87] "Starting device plugin registration server" Jul 09 17:39:32.748626 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.748606 2560 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 17:39:32.748664 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.748628 2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 17:39:32.748765 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.748736 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 17:39:32.748854 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.748837 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 17:39:32.748854 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.748850 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 17:39:32.749487 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.749467 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 17:39:32.749552 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.749515 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:32.805014 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.804968 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 17:39:32.806386 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.806314 2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 17:39:32.806386 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.806366 2560 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 17:39:32.806552 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.806399 2560 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 17:39:32.806552 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.806455 2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 17:39:32.808927 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.808905 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 17:39:32.849100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.849003 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 17:39:32.850295 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.850272 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientMemory" Jul 09 17:39:32.850413 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.850307 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 17:39:32.850413 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.850319 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientPID" Jul 09 17:39:32.850413 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.850345 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.877605 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.877576 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.877605 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.877609 2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-169.ec2.internal\": node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:32.906992 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.906943 2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal"] Jul 09 17:39:32.907157 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.907035 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 17:39:32.908531 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.908510 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientMemory" Jul 09 17:39:32.908599 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.908542 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 17:39:32.908599 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.908553 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientPID" Jul 09 17:39:32.910007 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.909993 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 17:39:32.910111 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.910094 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.910171 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.910127 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 17:39:32.910857 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.910835 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientMemory" Jul 09 17:39:32.910977 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.910874 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 17:39:32.910977 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.910886 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientPID" Jul 09 17:39:32.910977 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.910839 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientMemory" Jul 09 17:39:32.910977 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.910955 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 17:39:32.910977 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.910974 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientPID" Jul 09 17:39:32.911788 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.911762 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:32.912146 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.912125 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.912229 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.912165 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 17:39:32.913067 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.913052 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientMemory" Jul 09 17:39:32.913162 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.913082 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 17:39:32.913162 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.913092 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientPID" Jul 09 17:39:32.941346 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.941316 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-169.ec2.internal\" not found" node="ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.946091 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:32.946069 2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-169.ec2.internal\" not found" node="ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.979832 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.979787 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d136deba96e911acb4f180ecd1f1586-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal\" (UID: \"6d136deba96e911acb4f180ecd1f1586\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.979832 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.979831 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d136deba96e911acb4f180ecd1f1586-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal\" (UID: \"6d136deba96e911acb4f180ecd1f1586\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:32.980080 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:32.979850 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a18d5082e7af3d615bed4125b1e112d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.ec2.internal\" (UID: \"2a18d5082e7af3d615bed4125b1e112d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.012347 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.012315 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.080340 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.080306 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a18d5082e7af3d615bed4125b1e112d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.ec2.internal\" (UID: \"2a18d5082e7af3d615bed4125b1e112d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.080340 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.080341 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d136deba96e911acb4f180ecd1f1586-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal\" (UID: \"6d136deba96e911acb4f180ecd1f1586\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.080595 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.080359 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d136deba96e911acb4f180ecd1f1586-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal\" (UID: \"6d136deba96e911acb4f180ecd1f1586\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.080595 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.080409 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2a18d5082e7af3d615bed4125b1e112d-config\") pod \"kube-apiserver-proxy-ip-10-0-141-169.ec2.internal\" (UID: \"2a18d5082e7af3d615bed4125b1e112d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.080595 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.080444 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d136deba96e911acb4f180ecd1f1586-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal\" (UID: \"6d136deba96e911acb4f180ecd1f1586\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.080595 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.080409 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6d136deba96e911acb4f180ecd1f1586-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal\" (UID: \"6d136deba96e911acb4f180ecd1f1586\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.112461 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.112386 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.213249 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.213217 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.243483 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.243453 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.249051 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.249023 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" Jul 09 17:39:33.314340 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.314300 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.415035 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.414927 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.515467 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.515407 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.527948 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.527915 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 17:39:33.616494 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.616459 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.619608 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.619590 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 17:39:33.619783 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.619764 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 17:39:33.619854 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.619809 2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 17:39:33.716637 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.716596 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.722818 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.722782 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 17:34:32 +0000 UTC" deadline="2028-04-07 23:40:57.841655376 +0000 UTC" Jul 09 17:39:33.722917 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.722820 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15318h1m24.118839505s" Jul 09 17:39:33.783285 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.783245 2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 17:39:33.785000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.784982 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientMemory" Jul 09 17:39:33.785115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.785014 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 17:39:33.785115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.785024 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeHasSufficientPID" Jul 09 17:39:33.817214 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.817181 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.820064 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:33.820033 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a18d5082e7af3d615bed4125b1e112d.slice/crio-7fa549613fa74ea6e1fe445e96e4e8cb7ca437622698b886e613ff8103520366 WatchSource:0}: Error finding container 7fa549613fa74ea6e1fe445e96e4e8cb7ca437622698b886e613ff8103520366: Status 404 returned error can't find the container with id 7fa549613fa74ea6e1fe445e96e4e8cb7ca437622698b886e613ff8103520366 Jul 09 17:39:33.820602 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:33.820585 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d136deba96e911acb4f180ecd1f1586.slice/crio-daf99417708085d6227e0c76d80df9c37d253ebccddd6ab33dcef722de5e9191 WatchSource:0}: Error finding container daf99417708085d6227e0c76d80df9c37d253ebccddd6ab33dcef722de5e9191: Status 404 returned error can't find the container with id daf99417708085d6227e0c76d80df9c37d253ebccddd6ab33dcef722de5e9191 Jul 09 17:39:33.825972 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.825950 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 17:39:33.918290 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:33.918247 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:33.975813 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:33.975718 2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 17:39:34.019025 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.018982 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:34.119548 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.119499 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:34.220399 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.220357 2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-169.ec2.internal\" not found" Jul 09 17:39:34.231998 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.231915 2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 17:39:34.278124 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.278084 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" Jul 09 17:39:34.293919 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.293890 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 17:39:34.295257 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.295238 2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" Jul 09 17:39:34.304177 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.304150 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 17:39:34.605505 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.605406 2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 17:39:34.658387 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.658349 2560 apiserver.go:52] "Watching apiserver" Jul 09 17:39:34.669376 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.669335 2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 17:39:34.669969 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.669922 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8","openshift-image-registry/node-ca-4b8tl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal","openshift-multus/multus-additional-cni-plugins-rzjj6","openshift-multus/multus-l62j2","openshift-cluster-node-tuning-operator/tuned-4lhfl","openshift-dns/node-resolver-kzm26","openshift-multus/network-metrics-daemon-7vlf2","openshift-network-diagnostics/network-check-target-rdfxs","openshift-network-operator/iptables-alerter-sq4tk","openshift-ovn-kubernetes/ovnkube-node-8xxd4","kube-system/global-pull-secret-syncer-7wrgh","kube-system/konnectivity-agent-n2vlq"] Jul 09 17:39:34.671725 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.671699 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.673088 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.673056 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:34.674290 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.674099 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 17:39:34.674290 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.674228 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 17:39:34.674290 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.674257 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2v544\"" Jul 09 17:39:34.674678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.674645 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 17:39:34.674767 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.674702 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 17:39:34.674831 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.674773 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.674956 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.674934 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 17:39:34.675326 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.675303 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 17:39:34.675649 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.675631 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rmvhx\"" Jul 09 17:39:34.675790 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.675774 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 17:39:34.676307 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.676271 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:34.676408 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.676368 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:34.676992 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.676969 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 17:39:34.677239 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.677219 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 17:39:34.677311 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.677252 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 17:39:34.677515 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.677498 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jhzg5\"" Jul 09 17:39:34.677671 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.677653 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:34.677749 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.677724 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:34.677815 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.677799 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 17:39:34.679083 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.679060 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.680529 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.680507 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.682694 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.682188 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.682694 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.682568 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 17:39:34.682694 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.682662 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 17:39:34.682904 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.682769 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 17:39:34.682904 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.682862 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mts7w\"" Jul 09 17:39:34.682999 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.682958 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-87g9d\"" Jul 09 17:39:34.683719 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.683691 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.685521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.685055 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 17:39:34.685521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.685184 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.685761 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.685743 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 17:39:34.686205 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.686187 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kf7hl\"" Jul 09 17:39:34.686301 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.686289 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 17:39:34.686883 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.686439 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 17:39:34.686883 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.686493 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mlbtz\"" Jul 09 17:39:34.686883 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.686742 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 17:39:34.686883 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.686876 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.687123 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.686959 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:34.687433 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.687399 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 17:39:34.688033 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.687961 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-v5wk6\"" Jul 09 17:39:34.688033 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.687968 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 17:39:34.688289 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.688266 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 17:39:34.688365 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.688335 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 17:39:34.688481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.688465 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 17:39:34.688593 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.688578 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 17:39:34.688658 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.688598 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.690648 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690623 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.690752 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690661 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-conf-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.690752 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690696 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-os-release\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.690752 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690722 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-hostroot\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.690752 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690749 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.690964 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690774 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-netns\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.690964 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690805 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-kubelet\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.690964 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690829 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-daemon-config\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.690964 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690851 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-os-release\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.690964 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690877 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0114f7e-8bb8-40ce-a472-15872a204f8a-serviceca\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.690964 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:34.690964 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.690959 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-cnibin\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691299 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691019 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-cni-multus\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691299 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691052 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-multus-certs\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691299 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691056 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 17:39:34.691299 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691098 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 17:39:34.691299 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691155 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5r8k5\"" Jul 09 17:39:34.691299 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691233 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-system-cni-dir\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.691299 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691056 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 17:39:34.691299 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691283 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691326 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngxrh\" (UniqueName: \"kubernetes.io/projected/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-kube-api-access-ngxrh\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691352 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5287d7be-202c-41e9-99a8-5248fce6c989-agent-certs\") pod \"konnectivity-agent-n2vlq\" (UID: \"5287d7be-202c-41e9-99a8-5248fce6c989\") " pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691384 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-cni-binary-copy\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691407 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82tdk\" (UniqueName: \"kubernetes.io/projected/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-kube-api-access-82tdk\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691463 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691485 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0114f7e-8bb8-40ce-a472-15872a204f8a-host\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691536 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cnibin\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691555 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5287d7be-202c-41e9-99a8-5248fce6c989-konnectivity-ca\") pod \"konnectivity-agent-n2vlq\" (UID: \"5287d7be-202c-41e9-99a8-5248fce6c989\") " pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691589 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-cni-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691615 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-k8s-cni-cncf-io\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691638 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-socket-dir-parent\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691658 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-cni-bin\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691681 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79kr7\" (UniqueName: \"kubernetes.io/projected/a0114f7e-8bb8-40ce-a472-15872a204f8a-kube-api-access-79kr7\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.691709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691706 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccm2g\" (UniqueName: \"kubernetes.io/projected/86dbd8b7-a392-48da-901a-a23b1fabec38-kube-api-access-ccm2g\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:34.692361 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.691780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:34.692431 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.692394 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-system-cni-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.692490 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.692437 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-etc-kubernetes\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.694542 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.694445 2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 17:39:34.718143 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.718111 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-58bbl" Jul 09 17:39:34.730362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.730328 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-58bbl" Jul 09 17:39:34.779095 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.779059 2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 17:39:34.792737 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792691 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.792737 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792740 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-daemon-config\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.793000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792773 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-node-log\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.793000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792798 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-log-socket\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.793000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792869 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysctl-conf\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.793000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792890 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-systemd\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.793000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792909 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-os-release\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.793000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0114f7e-8bb8-40ce-a472-15872a204f8a-serviceca\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.793000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.792954 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-cnibin\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793002 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-multus-certs\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793014 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-cnibin\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793046 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-multus-certs\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793074 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-os-release\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-cni-bin\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793109 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-kubelet-config\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793132 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-dbus\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793158 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-cni-binary-copy\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.793306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793185 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82tdk\" (UniqueName: \"kubernetes.io/projected/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-kube-api-access-82tdk\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793368 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-slash\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793409 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-cni-netd\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793413 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-daemon-config\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793452 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovn-node-metrics-cert\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793506 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793547 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0114f7e-8bb8-40ce-a472-15872a204f8a-host\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793569 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a0114f7e-8bb8-40ce-a472-15872a204f8a-serviceca\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793581 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-tmp-dir\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-systemd\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793633 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a281f32e-f58f-45fd-936d-96e8de5c6121-iptables-alerter-script\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793650 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-modprobe-d\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793652 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a0114f7e-8bb8-40ce-a472-15872a204f8a-host\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.793678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793664 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysconfig\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793679 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-host\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793696 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-cni-binary-copy\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793716 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-cni-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-k8s-cni-cncf-io\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793762 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovnkube-script-lib\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793787 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-socket-dir-parent\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793814 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-cni-bin\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793821 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-k8s-cni-cncf-io\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793837 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79kr7\" (UniqueName: \"kubernetes.io/projected/a0114f7e-8bb8-40ce-a472-15872a204f8a-kube-api-access-79kr7\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793865 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccm2g\" (UniqueName: \"kubernetes.io/projected/86dbd8b7-a392-48da-901a-a23b1fabec38-kube-api-access-ccm2g\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793888 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-etc-kubernetes\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793894 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-socket-dir-parent\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793903 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-cni-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793912 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g4jd\" (UniqueName: \"kubernetes.io/projected/a281f32e-f58f-45fd-936d-96e8de5c6121-kube-api-access-7g4jd\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793903 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793938 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-lib-modules\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.794274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793935 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cni-binary-copy\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.793956 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-cni-bin\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794050 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794096 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-etc-kubernetes\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794103 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-systemd-units\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794129 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-var-lib-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794164 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794188 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-device-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794224 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-run-netns\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794226 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794258 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794283 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-kubernetes\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794305 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-sys\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794323 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-netns\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794353 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-run-netns\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794372 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-kubelet\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794391 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfjh8\" (UniqueName: \"kubernetes.io/projected/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-kube-api-access-dfjh8\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.795115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794446 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-registration-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794452 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-kubelet\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794492 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794510 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-cni-multus\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794546 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-host-var-lib-cni-multus\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.794584 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794590 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysctl-d\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794629 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-system-cni-dir\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.794657 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs podName:86dbd8b7-a392-48da-901a-a23b1fabec38 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:35.294636341 +0000 UTC m=+2.997137510 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs") pod "network-metrics-daemon-7vlf2" (UID: "86dbd8b7-a392-48da-901a-a23b1fabec38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794671 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-system-cni-dir\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794717 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794750 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngxrh\" (UniqueName: \"kubernetes.io/projected/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-kube-api-access-ngxrh\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794780 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5287d7be-202c-41e9-99a8-5248fce6c989-agent-certs\") pod \"konnectivity-agent-n2vlq\" (UID: \"5287d7be-202c-41e9-99a8-5248fce6c989\") " pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794826 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-kubelet\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794866 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a281f32e-f58f-45fd-936d-96e8de5c6121-host-slash\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794884 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-socket-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.795875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794904 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-sys-fs\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794940 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.794972 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-ovn\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795017 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwj72\" (UniqueName: \"kubernetes.io/projected/22257c40-5297-4b5a-b377-e7db21adfc99-kube-api-access-nwj72\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795059 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cnibin\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795091 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5287d7be-202c-41e9-99a8-5248fce6c989-konnectivity-ca\") pod \"konnectivity-agent-n2vlq\" (UID: \"5287d7be-202c-41e9-99a8-5248fce6c989\") " pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795114 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-cnibin\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795116 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-hosts-file\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795135 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795163 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovnkube-config\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795188 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-env-overrides\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795212 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-run\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795238 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795244 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795271 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-system-cni-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795372 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795375 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-system-cni-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.796541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795409 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795465 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8zf\" (UniqueName: \"kubernetes.io/projected/0115230c-4221-4545-ae1c-428cfa92d759-kube-api-access-mg8zf\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795496 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-var-lib-kubelet\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795526 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-conf-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795553 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22257c40-5297-4b5a-b377-e7db21adfc99-etc-tuned\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795566 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-multus-conf-dir\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795583 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22257c40-5297-4b5a-b377-e7db21adfc99-tmp\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795588 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5287d7be-202c-41e9-99a8-5248fce6c989-konnectivity-ca\") pod \"konnectivity-agent-n2vlq\" (UID: \"5287d7be-202c-41e9-99a8-5248fce6c989\") " pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795607 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-os-release\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795643 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-hostroot\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795672 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-os-release\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795716 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-hostroot\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795753 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-etc-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.797158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.795785 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slktw\" (UniqueName: \"kubernetes.io/projected/896c18d4-5355-4f6a-80c4-017b47ab37ad-kube-api-access-slktw\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.799020 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.799001 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5287d7be-202c-41e9-99a8-5248fce6c989-agent-certs\") pod \"konnectivity-agent-n2vlq\" (UID: \"5287d7be-202c-41e9-99a8-5248fce6c989\") " pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:34.804334 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.804305 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82tdk\" (UniqueName: \"kubernetes.io/projected/0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98-kube-api-access-82tdk\") pod \"multus-l62j2\" (UID: \"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98\") " pod="openshift-multus/multus-l62j2" Jul 09 17:39:34.806639 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.806563 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 17:39:34.806639 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.806592 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 17:39:34.806639 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.806607 2560 projected.go:196] Error preparing data for projected volume kube-api-access-5qdgs for pod openshift-network-diagnostics/network-check-target-rdfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:34.807116 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.806671 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs podName:3854aae9-5583-4345-a96c-9f3f2f90cbb3 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:35.306651483 +0000 UTC m=+3.009152658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5qdgs" (UniqueName: "kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs") pod "network-check-target-rdfxs" (UID: "3854aae9-5583-4345-a96c-9f3f2f90cbb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:34.808377 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.808349 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngxrh\" (UniqueName: \"kubernetes.io/projected/9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48-kube-api-access-ngxrh\") pod \"multus-additional-cni-plugins-rzjj6\" (UID: \"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48\") " pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.809278 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.809249 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79kr7\" (UniqueName: \"kubernetes.io/projected/a0114f7e-8bb8-40ce-a472-15872a204f8a-kube-api-access-79kr7\") pod \"node-ca-4b8tl\" (UID: \"a0114f7e-8bb8-40ce-a472-15872a204f8a\") " pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:34.809740 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.809716 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccm2g\" (UniqueName: \"kubernetes.io/projected/86dbd8b7-a392-48da-901a-a23b1fabec38-kube-api-access-ccm2g\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:34.812342 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.812280 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" event={"ID":"2a18d5082e7af3d615bed4125b1e112d","Type":"ContainerStarted","Data":"7fa549613fa74ea6e1fe445e96e4e8cb7ca437622698b886e613ff8103520366"} Jul 09 17:39:34.813552 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.813527 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" event={"ID":"6d136deba96e911acb4f180ecd1f1586","Type":"ContainerStarted","Data":"daf99417708085d6227e0c76d80df9c37d253ebccddd6ab33dcef722de5e9191"} Jul 09 17:39:34.896234 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896150 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.896234 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896193 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-ovn\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.896234 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896214 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwj72\" (UniqueName: \"kubernetes.io/projected/22257c40-5297-4b5a-b377-e7db21adfc99-kube-api-access-nwj72\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-hosts-file\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896264 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovnkube-config\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896272 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-ovn\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896270 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896288 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-env-overrides\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896397 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-run\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896481 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896507 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896532 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8zf\" (UniqueName: \"kubernetes.io/projected/0115230c-4221-4545-ae1c-428cfa92d759-kube-api-access-mg8zf\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896538 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-hosts-file\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.896548 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896556 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-var-lib-kubelet\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22257c40-5297-4b5a-b377-e7db21adfc99-etc-tuned\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896617 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22257c40-5297-4b5a-b377-e7db21adfc99-tmp\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.896633 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896659 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-etc-selinux\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896583 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-run\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896667 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-etc-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:34.896698 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret podName:c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:35.396675785 +0000 UTC m=+3.099176957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret") pod "global-pull-secret-syncer-7wrgh" (UID: "c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7") : object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896636 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-etc-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896717 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-var-lib-kubelet\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896737 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slktw\" (UniqueName: \"kubernetes.io/projected/896c18d4-5355-4f6a-80c4-017b47ab37ad-kube-api-access-slktw\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896768 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-node-log\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896794 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-log-socket\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896819 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysctl-conf\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896836 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-env-overrides\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896843 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-systemd\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896871 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-cni-bin\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896898 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-node-log\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896916 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovnkube-config\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896945 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-log-socket\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896970 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-systemd\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-kubelet-config\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897024 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-cni-bin\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897027 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-dbus\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897075 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-slash\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897098 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysctl-conf\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897099 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-cni-netd\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897139 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovn-node-metrics-cert\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897143 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-cni-netd\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897146 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-slash\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897147 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-dbus\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.896995 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-kubelet-config\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897164 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897203 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897208 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-tmp-dir\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.897953 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897234 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-systemd\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897270 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-run-systemd\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897283 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a281f32e-f58f-45fd-936d-96e8de5c6121-iptables-alerter-script\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897821 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-modprobe-d\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897868 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysconfig\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897924 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-host\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897957 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovnkube-script-lib\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.897995 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g4jd\" (UniqueName: \"kubernetes.io/projected/a281f32e-f58f-45fd-936d-96e8de5c6121-kube-api-access-7g4jd\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898025 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-lib-modules\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898081 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-systemd-units\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898110 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-var-lib-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898152 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898190 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-device-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898242 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-run-netns\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898271 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898526 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-tmp-dir\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.898776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898618 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-kubernetes\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899108 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a281f32e-f58f-45fd-936d-96e8de5c6121-iptables-alerter-script\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.899475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899260 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-modprobe-d\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899294 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-device-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.899475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899319 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.899475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899379 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.899475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899383 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/22257c40-5297-4b5a-b377-e7db21adfc99-etc-tuned\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.898307 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-kubernetes\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899459 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-var-lib-openvswitch\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899386 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-host\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899491 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-run-netns\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899506 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-sys\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899549 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfjh8\" (UniqueName: \"kubernetes.io/projected/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-kube-api-access-dfjh8\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899581 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-registration-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899618 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-systemd-units\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899634 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysctl-d\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899698 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-kubelet\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899720 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22257c40-5297-4b5a-b377-e7db21adfc99-tmp\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899758 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a281f32e-f58f-45fd-936d-96e8de5c6121-host-slash\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899780 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysconfig\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899794 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-sys\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.899817 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899809 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-socket-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899843 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a281f32e-f58f-45fd-936d-96e8de5c6121-host-slash\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899845 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-sys-fs\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.899866 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovn-node-metrics-cert\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.900009 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-socket-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.900016 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-sys-fs\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.900018 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/896c18d4-5355-4f6a-80c4-017b47ab37ad-ovnkube-script-lib\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.900083 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/896c18d4-5355-4f6a-80c4-017b47ab37ad-host-kubelet\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.900150 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0115230c-4221-4545-ae1c-428cfa92d759-registration-dir\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.900204 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-lib-modules\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.900362 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.900246 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/22257c40-5297-4b5a-b377-e7db21adfc99-etc-sysctl-d\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.908848 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.908815 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwj72\" (UniqueName: \"kubernetes.io/projected/22257c40-5297-4b5a-b377-e7db21adfc99-kube-api-access-nwj72\") pod \"tuned-4lhfl\" (UID: \"22257c40-5297-4b5a-b377-e7db21adfc99\") " pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:34.910896 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.910866 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g4jd\" (UniqueName: \"kubernetes.io/projected/a281f32e-f58f-45fd-936d-96e8de5c6121-kube-api-access-7g4jd\") pod \"iptables-alerter-sq4tk\" (UID: \"a281f32e-f58f-45fd-936d-96e8de5c6121\") " pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:34.911041 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.910866 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfjh8\" (UniqueName: \"kubernetes.io/projected/1c43ac19-a9e4-47c7-a4ca-a5d758033df8-kube-api-access-dfjh8\") pod \"node-resolver-kzm26\" (UID: \"1c43ac19-a9e4-47c7-a4ca-a5d758033df8\") " pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:34.912193 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.912171 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slktw\" (UniqueName: \"kubernetes.io/projected/896c18d4-5355-4f6a-80c4-017b47ab37ad-kube-api-access-slktw\") pod \"ovnkube-node-8xxd4\" (UID: \"896c18d4-5355-4f6a-80c4-017b47ab37ad\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:34.912522 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.912504 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8zf\" (UniqueName: \"kubernetes.io/projected/0115230c-4221-4545-ae1c-428cfa92d759-kube-api-access-mg8zf\") pod \"aws-ebs-csi-driver-node-zhks8\" (UID: \"0115230c-4221-4545-ae1c-428cfa92d759\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:34.985894 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.985857 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" Jul 09 17:39:34.994202 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:34.994173 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:35.006123 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.006088 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4b8tl" Jul 09 17:39:35.016246 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.016218 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l62j2" Jul 09 17:39:35.025216 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.025176 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" Jul 09 17:39:35.033320 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.033285 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kzm26" Jul 09 17:39:35.042098 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.042068 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sq4tk" Jul 09 17:39:35.050084 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.050056 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:39:35.056280 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.056246 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" Jul 09 17:39:35.302620 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.302585 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:35.302830 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:35.302773 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:35.302904 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:35.302871 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs podName:86dbd8b7-a392-48da-901a-a23b1fabec38 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:36.30284768 +0000 UTC m=+4.005348863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs") pod "network-metrics-daemon-7vlf2" (UID: "86dbd8b7-a392-48da-901a-a23b1fabec38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:35.403443 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.403392 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:35.403443 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.403448 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:35.403685 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:35.403548 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:35.403685 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:35.403576 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 17:39:35.403685 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:35.403601 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 17:39:35.403685 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:35.403617 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret podName:c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:36.403598256 +0000 UTC m=+4.106099426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret") pod "global-pull-secret-syncer-7wrgh" (UID: "c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7") : object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:35.403685 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:35.403650 2560 projected.go:196] Error preparing data for projected volume kube-api-access-5qdgs for pod openshift-network-diagnostics/network-check-target-rdfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:35.403913 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:35.403714 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs podName:3854aae9-5583-4345-a96c-9f3f2f90cbb3 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:36.403696774 +0000 UTC m=+4.106197956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qdgs" (UniqueName: "kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs") pod "network-check-target-rdfxs" (UID: "3854aae9-5583-4345-a96c-9f3f2f90cbb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:35.635903 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.635874 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3a4288_a9f1_444c_a3cd_e0bfd9f86e48.slice/crio-83374e65de7f368d276c1c84ffbcd8d50bf44b8631c024d99cce2027f023fdc3 WatchSource:0}: Error finding container 83374e65de7f368d276c1c84ffbcd8d50bf44b8631c024d99cce2027f023fdc3: Status 404 returned error can't find the container with id 83374e65de7f368d276c1c84ffbcd8d50bf44b8631c024d99cce2027f023fdc3 Jul 09 17:39:35.637687 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.637637 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda281f32e_f58f_45fd_936d_96e8de5c6121.slice/crio-6e37e8a9b1a98da68ace601af33bc05849e77466dc452a6153e9f985681b87cd WatchSource:0}: Error finding container 6e37e8a9b1a98da68ace601af33bc05849e77466dc452a6153e9f985681b87cd: Status 404 returned error can't find the container with id 6e37e8a9b1a98da68ace601af33bc05849e77466dc452a6153e9f985681b87cd Jul 09 17:39:35.641081 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.641055 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5287d7be_202c_41e9_99a8_5248fce6c989.slice/crio-3a40cae1dd23284af328c56f702455a5f72dfd072552c21aafc65d270afd373a WatchSource:0}: Error finding container 3a40cae1dd23284af328c56f702455a5f72dfd072552c21aafc65d270afd373a: Status 404 returned error can't find the container with id 3a40cae1dd23284af328c56f702455a5f72dfd072552c21aafc65d270afd373a Jul 09 17:39:35.642322 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.642299 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0115230c_4221_4545_ae1c_428cfa92d759.slice/crio-db96ae470dac22fea451037f1366654489293811a8caeec5361c61edb97c2d82 WatchSource:0}: Error finding container db96ae470dac22fea451037f1366654489293811a8caeec5361c61edb97c2d82: Status 404 returned error can't find the container with id db96ae470dac22fea451037f1366654489293811a8caeec5361c61edb97c2d82 Jul 09 17:39:35.643353 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.643305 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22257c40_5297_4b5a_b377_e7db21adfc99.slice/crio-4b892290eccddc9275cd7ba263a68d8b37b9e4b3c817306277c574aa54e48de3 WatchSource:0}: Error finding container 4b892290eccddc9275cd7ba263a68d8b37b9e4b3c817306277c574aa54e48de3: Status 404 returned error can't find the container with id 4b892290eccddc9275cd7ba263a68d8b37b9e4b3c817306277c574aa54e48de3 Jul 09 17:39:35.644263 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.644125 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbeb1a7_728c_4b64_aaba_5e1e3a1c1e98.slice/crio-e6278d6df47185859f545c966825ac753b9a23bb33b1f67de74e430724c9ca92 WatchSource:0}: Error finding container e6278d6df47185859f545c966825ac753b9a23bb33b1f67de74e430724c9ca92: Status 404 returned error can't find the container with id e6278d6df47185859f545c966825ac753b9a23bb33b1f67de74e430724c9ca92 Jul 09 17:39:35.646118 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.645715 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c43ac19_a9e4_47c7_a4ca_a5d758033df8.slice/crio-d8c68636256b1928fa28cc0d20e68d187859f79d28e3b4a4aecc488e6ddbbb8c WatchSource:0}: Error finding container d8c68636256b1928fa28cc0d20e68d187859f79d28e3b4a4aecc488e6ddbbb8c: Status 404 returned error can't find the container with id d8c68636256b1928fa28cc0d20e68d187859f79d28e3b4a4aecc488e6ddbbb8c Jul 09 17:39:35.646480 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.646433 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0114f7e_8bb8_40ce_a472_15872a204f8a.slice/crio-8efb897c0b649e21bf911284075473eb6fd4062c5460ffdcf975e896d4a3baa6 WatchSource:0}: Error finding container 8efb897c0b649e21bf911284075473eb6fd4062c5460ffdcf975e896d4a3baa6: Status 404 returned error can't find the container with id 8efb897c0b649e21bf911284075473eb6fd4062c5460ffdcf975e896d4a3baa6 Jul 09 17:39:35.647816 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:39:35.647761 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896c18d4_5355_4f6a_80c4_017b47ab37ad.slice/crio-768cbf3e672b4f33412ca4de7ddba0d115e190fd5362a4fcc967e660a066837a WatchSource:0}: Error finding container 768cbf3e672b4f33412ca4de7ddba0d115e190fd5362a4fcc967e660a066837a: Status 404 returned error can't find the container with id 768cbf3e672b4f33412ca4de7ddba0d115e190fd5362a4fcc967e660a066837a Jul 09 17:39:35.732948 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.732746 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:34 +0000 UTC" deadline="2028-03-22 17:29:09.967462832 +0000 UTC" Jul 09 17:39:35.732948 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.732949 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14927h49m34.234519112s" Jul 09 17:39:35.816830 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.816788 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzm26" event={"ID":"1c43ac19-a9e4-47c7-a4ca-a5d758033df8","Type":"ContainerStarted","Data":"d8c68636256b1928fa28cc0d20e68d187859f79d28e3b4a4aecc488e6ddbbb8c"} Jul 09 17:39:35.817928 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.817894 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" event={"ID":"22257c40-5297-4b5a-b377-e7db21adfc99","Type":"ContainerStarted","Data":"4b892290eccddc9275cd7ba263a68d8b37b9e4b3c817306277c574aa54e48de3"} Jul 09 17:39:35.818944 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.818911 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" event={"ID":"0115230c-4221-4545-ae1c-428cfa92d759","Type":"ContainerStarted","Data":"db96ae470dac22fea451037f1366654489293811a8caeec5361c61edb97c2d82"} Jul 09 17:39:35.820006 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.819979 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n2vlq" event={"ID":"5287d7be-202c-41e9-99a8-5248fce6c989","Type":"ContainerStarted","Data":"3a40cae1dd23284af328c56f702455a5f72dfd072552c21aafc65d270afd373a"} Jul 09 17:39:35.820979 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.820959 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sq4tk" event={"ID":"a281f32e-f58f-45fd-936d-96e8de5c6121","Type":"ContainerStarted","Data":"6e37e8a9b1a98da68ace601af33bc05849e77466dc452a6153e9f985681b87cd"} Jul 09 17:39:35.822725 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.822697 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerStarted","Data":"83374e65de7f368d276c1c84ffbcd8d50bf44b8631c024d99cce2027f023fdc3"} Jul 09 17:39:35.824364 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.824342 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"768cbf3e672b4f33412ca4de7ddba0d115e190fd5362a4fcc967e660a066837a"} Jul 09 17:39:35.825745 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.825722 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l62j2" event={"ID":"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98","Type":"ContainerStarted","Data":"e6278d6df47185859f545c966825ac753b9a23bb33b1f67de74e430724c9ca92"} Jul 09 17:39:35.827448 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.827407 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" event={"ID":"2a18d5082e7af3d615bed4125b1e112d","Type":"ContainerStarted","Data":"c5a364a9d40177b0e5eb1f353c0ea07feee5d051f0442055416641e650172c7b"} Jul 09 17:39:35.828482 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.828462 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4b8tl" event={"ID":"a0114f7e-8bb8-40ce-a472-15872a204f8a","Type":"ContainerStarted","Data":"8efb897c0b649e21bf911284075473eb6fd4062c5460ffdcf975e896d4a3baa6"} Jul 09 17:39:35.845372 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.845307 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-169.ec2.internal" podStartSLOduration=1.8452922680000001 podStartE2EDuration="1.845292268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:35.845071602 +0000 UTC m=+3.547572794" watchObservedRunningTime="2026-07-09 17:39:35.845292268 +0000 UTC m=+3.547793458" Jul 09 17:39:35.909537 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:35.909506 2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 17:39:36.311684 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.311640 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:36.311913 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.311890 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:36.312003 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.311968 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs podName:86dbd8b7-a392-48da-901a-a23b1fabec38 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:38.31194769 +0000 UTC m=+6.014448876 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs") pod "network-metrics-daemon-7vlf2" (UID: "86dbd8b7-a392-48da-901a-a23b1fabec38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:36.412363 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.412315 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:36.412363 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.412366 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:36.412632 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.412544 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:36.412632 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.412610 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret podName:c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:38.412592231 +0000 UTC m=+6.115093405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret") pod "global-pull-secret-syncer-7wrgh" (UID: "c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7") : object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:36.413063 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.413038 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 17:39:36.413165 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.413068 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 17:39:36.413165 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.413086 2560 projected.go:196] Error preparing data for projected volume kube-api-access-5qdgs for pod openshift-network-diagnostics/network-check-target-rdfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:36.413165 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.413132 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs podName:3854aae9-5583-4345-a96c-9f3f2f90cbb3 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:38.413116045 +0000 UTC m=+6.115617229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qdgs" (UniqueName: "kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs") pod "network-check-target-rdfxs" (UID: "3854aae9-5583-4345-a96c-9f3f2f90cbb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:36.733750 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.733677 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 17:34:34 +0000 UTC" deadline="2028-03-08 13:00:10.468724139 +0000 UTC" Jul 09 17:39:36.733750 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.733748 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14587h20m33.734981105s" Jul 09 17:39:36.806774 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.806741 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:36.806964 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.806877 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:36.807368 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.807343 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:36.807517 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.807483 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:36.807615 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.807551 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:36.807665 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:36.807615 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:36.841591 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.841513 2560 generic.go:356] "Generic (PLEG): container finished" podID="6d136deba96e911acb4f180ecd1f1586" containerID="51c07e5348421937377fe34820f4d7253f9acb3aa26dcb6c279c04051097e1f7" exitCode=0 Jul 09 17:39:36.842114 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:36.842089 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" event={"ID":"6d136deba96e911acb4f180ecd1f1586","Type":"ContainerDied","Data":"51c07e5348421937377fe34820f4d7253f9acb3aa26dcb6c279c04051097e1f7"} Jul 09 17:39:37.862974 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:37.862920 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" event={"ID":"6d136deba96e911acb4f180ecd1f1586","Type":"ContainerStarted","Data":"36c422d6ad10b0e6a83d4884f4906bfaaaa394f3ac3b67105a3536cccf06fbdd"} Jul 09 17:39:37.879372 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:37.879311 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-169.ec2.internal" podStartSLOduration=3.879287196 podStartE2EDuration="3.879287196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:37.879099233 +0000 UTC m=+5.581600433" watchObservedRunningTime="2026-07-09 17:39:37.879287196 +0000 UTC m=+5.581788387" Jul 09 17:39:38.328006 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:38.327353 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:38.328006 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.327566 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:38.328006 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.327632 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs podName:86dbd8b7-a392-48da-901a-a23b1fabec38 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:42.327613181 +0000 UTC m=+10.030114352 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs") pod "network-metrics-daemon-7vlf2" (UID: "86dbd8b7-a392-48da-901a-a23b1fabec38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:38.428371 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:38.428328 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:38.428559 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:38.428393 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:38.428636 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.428558 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:38.428636 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.428628 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret podName:c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:42.428608669 +0000 UTC m=+10.131109842 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret") pod "global-pull-secret-syncer-7wrgh" (UID: "c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7") : object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:38.429084 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.429063 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 17:39:38.429171 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.429091 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 17:39:38.429171 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.429105 2560 projected.go:196] Error preparing data for projected volume kube-api-access-5qdgs for pod openshift-network-diagnostics/network-check-target-rdfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:38.429171 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.429155 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs podName:3854aae9-5583-4345-a96c-9f3f2f90cbb3 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:42.429138893 +0000 UTC m=+10.131640084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qdgs" (UniqueName: "kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs") pod "network-check-target-rdfxs" (UID: "3854aae9-5583-4345-a96c-9f3f2f90cbb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:38.808447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:38.807227 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:38.808447 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.807373 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:38.808447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:38.807780 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:38.808447 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.807907 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:38.808447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:38.807956 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:38.808447 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:38.808032 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:40.807308 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:40.806727 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:40.807308 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:40.806875 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:40.807308 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:40.806887 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:40.807308 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:40.806997 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:40.807308 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:40.807069 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:40.807308 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:40.807158 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:42.365966 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:42.365616 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:42.365966 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.365774 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:42.365966 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.365855 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs podName:86dbd8b7-a392-48da-901a-a23b1fabec38 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:50.365832271 +0000 UTC m=+18.068333440 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs") pod "network-metrics-daemon-7vlf2" (UID: "86dbd8b7-a392-48da-901a-a23b1fabec38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:42.466140 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:42.465992 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:42.466140 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:42.466049 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:42.466140 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.466145 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 17:39:42.466473 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.466169 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 17:39:42.466473 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.466177 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:42.466473 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.466182 2560 projected.go:196] Error preparing data for projected volume kube-api-access-5qdgs for pod openshift-network-diagnostics/network-check-target-rdfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:42.466473 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.466234 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs podName:3854aae9-5583-4345-a96c-9f3f2f90cbb3 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:50.466215034 +0000 UTC m=+18.168716209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qdgs" (UniqueName: "kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs") pod "network-check-target-rdfxs" (UID: "3854aae9-5583-4345-a96c-9f3f2f90cbb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:42.466473 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.466254 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret podName:c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7 nodeName:}" failed. No retries permitted until 2026-07-09 17:39:50.466244351 +0000 UTC m=+18.168745522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret") pod "global-pull-secret-syncer-7wrgh" (UID: "c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7") : object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:42.808042 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:42.808002 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:42.808242 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.808145 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:42.808242 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:42.808213 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:42.808369 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.808336 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:42.808439 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:42.808376 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:42.808491 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:42.808458 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:44.807378 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:44.807340 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:44.808345 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:44.808306 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:44.808626 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:44.808596 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:44.808681 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:44.808175 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:44.808681 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:44.808650 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:44.809258 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:44.809226 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:46.807622 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:46.807583 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:46.808110 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:46.807583 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:46.808110 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:46.807736 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:46.808110 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:46.807591 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:46.808110 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:46.807802 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:46.808110 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:46.807888 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:48.807227 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:48.807192 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:48.807227 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:48.807222 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:48.807766 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:48.807199 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:48.807766 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:48.807332 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:48.807766 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:48.807466 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:48.807766 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:48.807541 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:50.426641 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:50.426601 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:50.427091 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.426762 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:50.427091 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.426834 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs podName:86dbd8b7-a392-48da-901a-a23b1fabec38 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:06.426817613 +0000 UTC m=+34.129318781 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs") pod "network-metrics-daemon-7vlf2" (UID: "86dbd8b7-a392-48da-901a-a23b1fabec38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:39:50.527355 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:50.527309 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:50.527355 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:50.527354 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:50.527572 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.527498 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:50.527572 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.527518 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 17:39:50.527572 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.527536 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 17:39:50.527572 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.527549 2560 projected.go:196] Error preparing data for projected volume kube-api-access-5qdgs for pod openshift-network-diagnostics/network-check-target-rdfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:50.527572 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.527569 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret podName:c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:06.527550464 +0000 UTC m=+34.230051644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret") pod "global-pull-secret-syncer-7wrgh" (UID: "c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7") : object "kube-system"/"original-pull-secret" not registered Jul 09 17:39:50.527735 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.527584 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs podName:3854aae9-5583-4345-a96c-9f3f2f90cbb3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:06.527575541 +0000 UTC m=+34.230076709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qdgs" (UniqueName: "kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs") pod "network-check-target-rdfxs" (UID: "3854aae9-5583-4345-a96c-9f3f2f90cbb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:39:50.807235 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:50.807196 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:50.807481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:50.807199 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:50.807481 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.807347 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:50.807481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:50.807360 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:50.807481 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.807456 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:50.807722 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:50.807540 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:52.807633 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:52.807591 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:52.808105 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:52.807712 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:52.808105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:52.807870 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:52.808240 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:52.808219 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:52.809877 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:52.808383 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:52.809877 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:52.808534 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:54.807632 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:54.806923 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:54.807632 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:54.807049 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:54.807632 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:54.807483 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:54.807632 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:54.807592 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:54.808382 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:54.808241 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:54.808382 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:54.808340 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:54.899830 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:54.899787 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" event={"ID":"0115230c-4221-4545-ae1c-428cfa92d759","Type":"ContainerStarted","Data":"e443c5f234feb5d94051bd996321a767b5009564313b7246e948c80ee5c66a8d"} Jul 09 17:39:54.902030 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:54.901996 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n2vlq" event={"ID":"5287d7be-202c-41e9-99a8-5248fce6c989","Type":"ContainerStarted","Data":"0aa5014efc59821c36d80dec529b47dc104e0b7798ccef53d9c3f66b03a98f2b"} Jul 09 17:39:54.903639 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:54.903608 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerStarted","Data":"90f05e56cf0c61b0a71220569862512682ba6eed4f57c08a16a79edc5a5625eb"} Jul 09 17:39:54.921068 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:54.920932 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n2vlq" podStartSLOduration=12.388809441 podStartE2EDuration="22.920909007s" totalImagesPullingTime="10.532099566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:54.919911183 +0000 UTC m=+22.622412374" watchObservedRunningTime="2026-07-09 17:39:54.920909007 +0000 UTC m=+22.623410274" Jul 09 17:39:55.904875 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.904664 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:55.905707 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.905336 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:55.907855 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.907823 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l62j2" event={"ID":"0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98","Type":"ContainerStarted","Data":"9781aae9b76a4949eea7d3adb1bd408e74381fc8e316a7d5f64916b687d314e2"} Jul 09 17:39:55.909289 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.909263 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4b8tl" event={"ID":"a0114f7e-8bb8-40ce-a472-15872a204f8a","Type":"ContainerStarted","Data":"1bc36b3b81ed74267724db28ea8592cd3db7b6947d55752241ced78f1f450b65"} Jul 09 17:39:55.910627 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.910607 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kzm26" event={"ID":"1c43ac19-a9e4-47c7-a4ca-a5d758033df8","Type":"ContainerStarted","Data":"55feb96cde38109fcc044871bcc954f99fc45184ea77a314b65e059d267bdff6"} Jul 09 17:39:55.911944 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.911920 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" event={"ID":"22257c40-5297-4b5a-b377-e7db21adfc99","Type":"ContainerStarted","Data":"6697bf382497af18501ebc667479e6b55e5b6f4f8a4bd4871297ef4aa10ec9c7"} Jul 09 17:39:55.913184 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.913165 2560 generic.go:356] "Generic (PLEG): container finished" podID="9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48" containerID="90f05e56cf0c61b0a71220569862512682ba6eed4f57c08a16a79edc5a5625eb" exitCode=0 Jul 09 17:39:55.913270 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.913245 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerDied","Data":"90f05e56cf0c61b0a71220569862512682ba6eed4f57c08a16a79edc5a5625eb"} Jul 09 17:39:55.916281 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.916255 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"77ee131deba6bb3475ad73ef9343f68a44dbb3e4195354e3d449cea3fd221b28"} Jul 09 17:39:55.916384 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.916325 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"32a65b71a3f8206dd27d7b7f8187d34e42d4f81016f05ddfbd79e6040231b32b"} Jul 09 17:39:55.916384 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.916371 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:55.916485 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.916385 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"1d189be64b19d4ddb34859cf0bdf44c62ecda73f41be8b173173be78eea47620"} Jul 09 17:39:55.916485 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.916431 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"5d5916edebb066b5b51ff6c2d054a139b4c9cebe0404b1fe5bcb90d2cb24d6b0"} Jul 09 17:39:55.916485 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.916465 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"be6f991f99e36b400b03b2574bfb88000ce87a20c88b016a23c7bec3a7a69cd7"} Jul 09 17:39:55.916583 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.916493 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"442bfa1333e1d429c321df3c0b764a3c7f2a6b5d059088bc1093e4e38ba0c89c"} Jul 09 17:39:55.916908 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.916890 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n2vlq" Jul 09 17:39:55.936263 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.936209 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4lhfl" podStartSLOduration=3.901004795 podStartE2EDuration="22.936192724s" totalImagesPullingTime="19.035187929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:55.935997736 +0000 UTC m=+23.638498927" watchObservedRunningTime="2026-07-09 17:39:55.936192724 +0000 UTC m=+23.638693914" Jul 09 17:39:55.949700 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:55.949652 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kzm26" podStartSLOduration=3.918479124 podStartE2EDuration="22.949633259s" totalImagesPullingTime="19.031154135s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:55.949227862 +0000 UTC m=+23.651729052" watchObservedRunningTime="2026-07-09 17:39:55.949633259 +0000 UTC m=+23.652134448" Jul 09 17:39:56.008639 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.008584 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4b8tl" podStartSLOduration=4.978802777 podStartE2EDuration="24.008567609s" totalImagesPullingTime="19.029764832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:56.008083156 +0000 UTC m=+23.710584368" watchObservedRunningTime="2026-07-09 17:39:56.008567609 +0000 UTC m=+23.711068864" Jul 09 17:39:56.024270 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.024217 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l62j2" podStartSLOduration=3.947134759 podStartE2EDuration="23.024196966s" totalImagesPullingTime="19.077062207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:56.023622264 +0000 UTC m=+23.726123454" watchObservedRunningTime="2026-07-09 17:39:56.024196966 +0000 UTC m=+23.726698157" Jul 09 17:39:56.251591 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.251558 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 17:39:56.761402 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.761281 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T17:39:56.251581427Z","UUID":"097dc8a7-eed2-4d86-969c-f01c0a4b5a48","Handler":null,"Name":"","Endpoint":""} Jul 09 17:39:56.763468 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.763441 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 17:39:56.763636 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.763485 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 17:39:56.807451 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.807400 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:56.807646 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.807436 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:56.807646 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:56.807571 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:56.807776 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:56.807641 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:56.807776 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.807706 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:56.807898 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:56.807789 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:39:56.920759 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.920707 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" event={"ID":"0115230c-4221-4545-ae1c-428cfa92d759","Type":"ContainerStarted","Data":"9903c9ba839d676e88e060f430b8f680625ce6f4528413f6ea9e0b056dda5d31"} Jul 09 17:39:56.922478 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.922448 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sq4tk" event={"ID":"a281f32e-f58f-45fd-936d-96e8de5c6121","Type":"ContainerStarted","Data":"3955a918760f5d9610168f5c4656ba2c17d8986d93d481570f91165c457f4e6e"} Jul 09 17:39:56.937624 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:56.937558 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sq4tk" podStartSLOduration=4.898229144 podStartE2EDuration="23.937539959s" totalImagesPullingTime="19.039310815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:56.936964168 +0000 UTC m=+24.639465359" watchObservedRunningTime="2026-07-09 17:39:56.937539959 +0000 UTC m=+24.640041149" Jul 09 17:39:57.927709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:57.927269 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" event={"ID":"0115230c-4221-4545-ae1c-428cfa92d759","Type":"ContainerStarted","Data":"e9a6e68f3c04081807a7d46b77a739d49f0c206001f40ca914c7f0ff73b85eb4"} Jul 09 17:39:57.931373 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:57.931291 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"3e59186e23da282cf08da8227a2ff005814db3163aee2ea5337ec7be8e915fc0"} Jul 09 17:39:57.945703 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:57.945630 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zhks8" podStartSLOduration=3.717644128 podStartE2EDuration="24.945611724s" totalImagesPullingTime="21.227967596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:39:57.945000769 +0000 UTC m=+25.647501959" watchObservedRunningTime="2026-07-09 17:39:57.945611724 +0000 UTC m=+25.648112914" Jul 09 17:39:58.807222 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:58.807178 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:39:58.807452 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:58.807178 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:39:58.807452 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:39:58.807342 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:39:58.807452 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:58.807336 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:39:58.807619 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:58.807465 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:39:58.807619 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:39:58.807542 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:40:00.807319 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.807106 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:00.808138 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.807118 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:00.808138 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:00.807400 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:40:00.808138 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:00.807487 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:40:00.808138 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.807160 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:40:00.808138 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:00.807563 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:40:00.939577 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.939543 2560 generic.go:356] "Generic (PLEG): container finished" podID="9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48" containerID="5c0344d4b862c35e2e41ca0e99b3d58c9ca2b26fce23a404b534992586df5fd7" exitCode=0 Jul 09 17:40:00.939760 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.939626 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerDied","Data":"5c0344d4b862c35e2e41ca0e99b3d58c9ca2b26fce23a404b534992586df5fd7"} Jul 09 17:40:00.943556 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.943531 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" event={"ID":"896c18d4-5355-4f6a-80c4-017b47ab37ad","Type":"ContainerStarted","Data":"65840644ef861897f7c12c4649bb755823e1e42cd721f6e84c2c1ff12799d14e"} Jul 09 17:40:00.943874 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.943851 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:40:00.961565 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.961532 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:40:00.993882 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:00.993826 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" podStartSLOduration=8.752807635 podStartE2EDuration="27.99381064s" totalImagesPullingTime="19.241003005s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:00.991288844 +0000 UTC m=+28.693790033" watchObservedRunningTime="2026-07-09 17:40:00.99381064 +0000 UTC m=+28.696311829" Jul 09 17:40:01.945908 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.945880 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:40:01.946461 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.945957 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:40:01.961604 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.961575 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:40:01.975360 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.975331 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7wrgh"] Jul 09 17:40:01.975543 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.975468 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:01.975603 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:01.975566 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:40:01.978385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.978357 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vlf2"] Jul 09 17:40:01.978533 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.978511 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:40:01.978631 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:01.978612 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:40:01.988280 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.988253 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rdfxs"] Jul 09 17:40:01.988453 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:01.988360 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:01.988535 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:01.988459 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:40:02.949574 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:02.949529 2560 generic.go:356] "Generic (PLEG): container finished" podID="9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48" containerID="805c5d91f334cad457188ea50c185da6b63c1129762972c9215e5b6e5fb91811" exitCode=0 Jul 09 17:40:02.949999 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:02.949607 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerDied","Data":"805c5d91f334cad457188ea50c185da6b63c1129762972c9215e5b6e5fb91811"} Jul 09 17:40:03.807133 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:03.807091 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:40:03.807133 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:03.807125 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:03.807367 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:03.807127 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:03.807367 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:03.807211 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:40:03.807367 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:03.807338 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:40:03.807540 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:03.807457 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:40:04.956535 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:04.956299 2560 generic.go:356] "Generic (PLEG): container finished" podID="9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48" containerID="49541e0527560158510a21b9b0559f11ba8f6261f7bdabf520caec43b76f865c" exitCode=0 Jul 09 17:40:04.956981 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:04.956381 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerDied","Data":"49541e0527560158510a21b9b0559f11ba8f6261f7bdabf520caec43b76f865c"} Jul 09 17:40:05.807593 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:05.807554 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:05.807593 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:05.807589 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:05.807887 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:05.807561 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:40:05.807887 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:05.807695 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-7wrgh" podUID="c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7" Jul 09 17:40:05.807887 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:05.807793 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7vlf2" podUID="86dbd8b7-a392-48da-901a-a23b1fabec38" Jul 09 17:40:05.807887 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:05.807879 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rdfxs" podUID="3854aae9-5583-4345-a96c-9f3f2f90cbb3" Jul 09 17:40:06.455255 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:06.455216 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:40:06.455860 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:06.455394 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:40:06.455860 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:06.455482 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs podName:86dbd8b7-a392-48da-901a-a23b1fabec38 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:38.455464254 +0000 UTC m=+66.157965422 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs") pod "network-metrics-daemon-7vlf2" (UID: "86dbd8b7-a392-48da-901a-a23b1fabec38") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 17:40:06.556271 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:06.556230 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:06.556271 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:06.556268 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:06.556486 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:06.556398 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 17:40:06.556486 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:06.556429 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 17:40:06.556486 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:06.556437 2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 17:40:06.556486 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:06.556449 2560 projected.go:196] Error preparing data for projected volume kube-api-access-5qdgs for pod openshift-network-diagnostics/network-check-target-rdfxs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:40:06.556610 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:06.556482 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret podName:c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:38.556468482 +0000 UTC m=+66.258969650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret") pod "global-pull-secret-syncer-7wrgh" (UID: "c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7") : object "kube-system"/"original-pull-secret" not registered Jul 09 17:40:06.556610 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:06.556506 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs podName:3854aae9-5583-4345-a96c-9f3f2f90cbb3 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:38.556499673 +0000 UTC m=+66.259000842 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5qdgs" (UniqueName: "kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs") pod "network-check-target-rdfxs" (UID: "3854aae9-5583-4345-a96c-9f3f2f90cbb3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 17:40:07.625301 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.625212 2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-169.ec2.internal" event="NodeReady" Jul 09 17:40:07.625818 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.625375 2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 17:40:07.697674 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.697636 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-94464db7-rhhq7"] Jul 09 17:40:07.699720 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.699682 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.703051 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.703025 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c6j4v\"" Jul 09 17:40:07.703605 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.703587 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 17:40:07.703755 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.703735 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 17:40:07.703811 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.703765 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 17:40:07.716075 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.716046 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 17:40:07.725072 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.725042 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-94464db7-rhhq7"] Jul 09 17:40:07.745374 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.745336 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6dtwp"] Jul 09 17:40:07.747395 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.747366 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.748646 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.748604 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mcntj"] Jul 09 17:40:07.750323 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.750301 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mcntj" Jul 09 17:40:07.755091 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.755064 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 17:40:07.755491 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.755466 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lljhl\"" Jul 09 17:40:07.756111 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.755717 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 17:40:07.756111 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.755792 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nnsbc\"" Jul 09 17:40:07.756111 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.755977 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 17:40:07.757690 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.757668 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 17:40:07.757803 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.757756 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 17:40:07.765338 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.765308 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-installation-pull-secrets\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.765471 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.765345 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-bound-sa-token\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.765471 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.765449 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.765587 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.765473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-trusted-ca\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.765587 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.765554 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/947050c5-395e-4330-8557-271ad164c107-ca-trust-extracted\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.765687 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.765593 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-registry-certificates\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.765687 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.765619 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-image-registry-private-configuration\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.765687 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.765637 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5smj4\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-kube-api-access-5smj4\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.776935 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.776886 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6dtwp"] Jul 09 17:40:07.777572 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.777540 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mcntj"] Jul 09 17:40:07.807248 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.807215 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:07.807414 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.807215 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:07.807414 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.807215 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:40:07.811433 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.811392 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 17:40:07.811562 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.811457 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 17:40:07.811562 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.811461 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f99f5\"" Jul 09 17:40:07.811562 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.811548 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 17:40:07.811697 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.811461 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 17:40:07.811697 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.811681 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9rs28\"" Jul 09 17:40:07.866803 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.866762 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-image-registry-private-configuration\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.866803 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.866806 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5smj4\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-kube-api-access-5smj4\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.867059 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.866840 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40f09dde-c76e-410b-98cd-cdb1de2858d4-config-volume\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.867059 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.866993 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-installation-pull-secrets\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.867059 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867045 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-bound-sa-token\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.867204 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867108 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.867204 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867141 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.867204 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867168 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-trusted-ca\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.867331 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:07.867259 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 17:40:07.867331 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:07.867279 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94464db7-rhhq7: secret "image-registry-tls" not found Jul 09 17:40:07.867331 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867259 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f09dde-c76e-410b-98cd-cdb1de2858d4-tmp-dir\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.867509 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:07.867448 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls podName:947050c5-395e-4330-8557-271ad164c107 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.367405425 +0000 UTC m=+36.069906594 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls") pod "image-registry-94464db7-rhhq7" (UID: "947050c5-395e-4330-8557-271ad164c107") : secret "image-registry-tls" not found Jul 09 17:40:07.867509 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5z4\" (UniqueName: \"kubernetes.io/projected/40f09dde-c76e-410b-98cd-cdb1de2858d4-kube-api-access-tw5z4\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.867509 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867504 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmbcg\" (UniqueName: \"kubernetes.io/projected/aca88050-a7a2-4388-b3bd-20c66b173963-kube-api-access-wmbcg\") pod \"ingress-canary-mcntj\" (UID: \"aca88050-a7a2-4388-b3bd-20c66b173963\") " pod="openshift-ingress-canary/ingress-canary-mcntj" Jul 09 17:40:07.867679 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867541 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aca88050-a7a2-4388-b3bd-20c66b173963-cert\") pod \"ingress-canary-mcntj\" (UID: \"aca88050-a7a2-4388-b3bd-20c66b173963\") " pod="openshift-ingress-canary/ingress-canary-mcntj" Jul 09 17:40:07.867679 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867579 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/947050c5-395e-4330-8557-271ad164c107-ca-trust-extracted\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.867679 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867643 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-registry-certificates\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.867947 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.867928 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/947050c5-395e-4330-8557-271ad164c107-ca-trust-extracted\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.868224 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.868196 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-registry-certificates\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.868319 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.868300 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-trusted-ca\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.872152 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.872124 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-installation-pull-secrets\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.872152 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.872147 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-image-registry-private-configuration\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.882708 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.882633 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5smj4\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-kube-api-access-5smj4\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.887945 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.887510 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-bound-sa-token\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:07.961521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.961484 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:07.968167 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.968137 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f09dde-c76e-410b-98cd-cdb1de2858d4-tmp-dir\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.968314 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.968177 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5z4\" (UniqueName: \"kubernetes.io/projected/40f09dde-c76e-410b-98cd-cdb1de2858d4-kube-api-access-tw5z4\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.968314 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.968198 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmbcg\" (UniqueName: \"kubernetes.io/projected/aca88050-a7a2-4388-b3bd-20c66b173963-kube-api-access-wmbcg\") pod \"ingress-canary-mcntj\" (UID: \"aca88050-a7a2-4388-b3bd-20c66b173963\") " pod="openshift-ingress-canary/ingress-canary-mcntj" Jul 09 17:40:07.968434 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.968348 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aca88050-a7a2-4388-b3bd-20c66b173963-cert\") pod \"ingress-canary-mcntj\" (UID: \"aca88050-a7a2-4388-b3bd-20c66b173963\") " pod="openshift-ingress-canary/ingress-canary-mcntj" Jul 09 17:40:07.968488 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.968448 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40f09dde-c76e-410b-98cd-cdb1de2858d4-config-volume\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.968546 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.968499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.968610 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:07.968595 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 17:40:07.968859 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:07.968666 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls podName:40f09dde-c76e-410b-98cd-cdb1de2858d4 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:08.468647944 +0000 UTC m=+36.171149114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls") pod "dns-default-6dtwp" (UID: "40f09dde-c76e-410b-98cd-cdb1de2858d4") : secret "dns-default-metrics-tls" not found Jul 09 17:40:07.968859 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.968593 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40f09dde-c76e-410b-98cd-cdb1de2858d4-tmp-dir\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.969054 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.969035 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/40f09dde-c76e-410b-98cd-cdb1de2858d4-config-volume\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:07.971227 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.971199 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aca88050-a7a2-4388-b3bd-20c66b173963-cert\") pod \"ingress-canary-mcntj\" (UID: \"aca88050-a7a2-4388-b3bd-20c66b173963\") " pod="openshift-ingress-canary/ingress-canary-mcntj" Jul 09 17:40:07.989067 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.989030 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmbcg\" (UniqueName: \"kubernetes.io/projected/aca88050-a7a2-4388-b3bd-20c66b173963-kube-api-access-wmbcg\") pod \"ingress-canary-mcntj\" (UID: \"aca88050-a7a2-4388-b3bd-20c66b173963\") " pod="openshift-ingress-canary/ingress-canary-mcntj" Jul 09 17:40:07.991309 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:07.991283 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5z4\" (UniqueName: \"kubernetes.io/projected/40f09dde-c76e-410b-98cd-cdb1de2858d4-kube-api-access-tw5z4\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:08.066702 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:08.066661 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mcntj" Jul 09 17:40:08.127011 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:08.126947 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:08.227500 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:08.227202 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mcntj"] Jul 09 17:40:08.260436 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:08.260378 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:08.372588 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:08.372545 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:08.372802 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:08.372713 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 17:40:08.372802 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:08.372739 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94464db7-rhhq7: secret "image-registry-tls" not found Jul 09 17:40:08.372802 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:08.372805 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls podName:947050c5-395e-4330-8557-271ad164c107 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:09.372784492 +0000 UTC m=+37.075285679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls") pod "image-registry-94464db7-rhhq7" (UID: "947050c5-395e-4330-8557-271ad164c107") : secret "image-registry-tls" not found Jul 09 17:40:08.473064 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:08.473023 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:08.473291 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:08.473187 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 17:40:08.473291 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:08.473274 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls podName:40f09dde-c76e-410b-98cd-cdb1de2858d4 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:09.473252153 +0000 UTC m=+37.175753333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls") pod "dns-default-6dtwp" (UID: "40f09dde-c76e-410b-98cd-cdb1de2858d4") : secret "dns-default-metrics-tls" not found Jul 09 17:40:08.829049 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:08.828967 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:08.965133 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:08.965088 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:09.249097 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.249029 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:09.377385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.377337 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:09.382387 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.382357 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:09.382573 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:09.382540 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 17:40:09.382573 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:09.382556 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94464db7-rhhq7: secret "image-registry-tls" not found Jul 09 17:40:09.382652 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:09.382611 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls podName:947050c5-395e-4330-8557-271ad164c107 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:11.382596571 +0000 UTC m=+39.085097738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls") pod "image-registry-94464db7-rhhq7" (UID: "947050c5-395e-4330-8557-271ad164c107") : secret "image-registry-tls" not found Jul 09 17:40:09.483482 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.483445 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:09.483656 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:09.483625 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 17:40:09.483733 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:09.483703 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls podName:40f09dde-c76e-410b-98cd-cdb1de2858d4 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:11.483683996 +0000 UTC m=+39.186185165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls") pod "dns-default-6dtwp" (UID: "40f09dde-c76e-410b-98cd-cdb1de2858d4") : secret "dns-default-metrics-tls" not found Jul 09 17:40:09.531686 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.531588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:09.609737 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:40:09.609699 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca88050_a7a2_4388_b3bd_20c66b173963.slice/crio-bc88dfb37365cb05abdf171208dcffc3f52b91c956e4753c290eb6d4947ca717 WatchSource:0}: Error finding container bc88dfb37365cb05abdf171208dcffc3f52b91c956e4753c290eb6d4947ca717: Status 404 returned error can't find the container with id bc88dfb37365cb05abdf171208dcffc3f52b91c956e4753c290eb6d4947ca717 Jul 09 17:40:09.651197 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.651177 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:09.801412 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.801317 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:09.969768 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.969728 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mcntj" event={"ID":"aca88050-a7a2-4388-b3bd-20c66b173963","Type":"ContainerStarted","Data":"bc88dfb37365cb05abdf171208dcffc3f52b91c956e4753c290eb6d4947ca717"} Jul 09 17:40:09.972518 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.972479 2560 generic.go:356] "Generic (PLEG): container finished" podID="9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48" containerID="ad851ec99a102dc39726ed984863f846e3eef1fb0d858f59649edaacc6d9b89d" exitCode=0 Jul 09 17:40:09.972630 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:09.972541 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerDied","Data":"ad851ec99a102dc39726ed984863f846e3eef1fb0d858f59649edaacc6d9b89d"} Jul 09 17:40:10.266151 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:10.266116 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:10.453124 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:10.453094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:10.554458 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:10.554404 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:10.977713 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:10.977678 2560 generic.go:356] "Generic (PLEG): container finished" podID="9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48" containerID="f0b0b70c91d6e8e170a0a188276573b67755ae7272742faeeb770c0c8df58723" exitCode=0 Jul 09 17:40:10.978209 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:10.977736 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerDied","Data":"f0b0b70c91d6e8e170a0a188276573b67755ae7272742faeeb770c0c8df58723"} Jul 09 17:40:11.401549 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:11.401434 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:11.401744 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:11.401623 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 17:40:11.401744 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:11.401646 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94464db7-rhhq7: secret "image-registry-tls" not found Jul 09 17:40:11.401744 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:11.401720 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls podName:947050c5-395e-4330-8557-271ad164c107 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:15.401695879 +0000 UTC m=+43.104197064 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls") pod "image-registry-94464db7-rhhq7" (UID: "947050c5-395e-4330-8557-271ad164c107") : secret "image-registry-tls" not found Jul 09 17:40:11.502789 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:11.502748 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:11.503031 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:11.502940 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 17:40:11.503031 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:11.503029 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls podName:40f09dde-c76e-410b-98cd-cdb1de2858d4 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:15.503005738 +0000 UTC m=+43.205506911 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls") pod "dns-default-6dtwp" (UID: "40f09dde-c76e-410b-98cd-cdb1de2858d4") : secret "dns-default-metrics-tls" not found Jul 09 17:40:11.984011 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:11.983976 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" event={"ID":"9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48","Type":"ContainerStarted","Data":"68955f212fd0993f287a806300bdb0056b73a0eb36a9ab970e9395ae01d7a340"} Jul 09 17:40:12.011659 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:12.011606 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rzjj6" podStartSLOduration=9.700183899 podStartE2EDuration="40.011587957s" totalImagesPullingTime="30.311404058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:12.010931989 +0000 UTC m=+39.713433191" watchObservedRunningTime="2026-07-09 17:40:12.011587957 +0000 UTC m=+39.714089146" Jul 09 17:40:12.379719 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:12.379410 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:12.476970 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:12.476926 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:12.577167 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:12.577140 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:12.987561 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:12.987518 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mcntj" event={"ID":"aca88050-a7a2-4388-b3bd-20c66b173963","Type":"ContainerStarted","Data":"6b1fa3dbe487b2b4c7b9f70589fb170efd6204ee6756a1a1c17af03b72297f99"} Jul 09 17:40:13.008217 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:13.008164 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mcntj" podStartSLOduration=3.4837964169999998 podStartE2EDuration="6.00814819s" totalImagesPullingTime="2.524351773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:13.007509942 +0000 UTC m=+40.710011133" watchObservedRunningTime="2026-07-09 17:40:13.00814819 +0000 UTC m=+40.710649379" Jul 09 17:40:14.282409 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:14.282371 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:15.434216 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:15.434171 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:15.434594 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:15.434324 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 17:40:15.434594 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:15.434340 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94464db7-rhhq7: secret "image-registry-tls" not found Jul 09 17:40:15.434594 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:15.434399 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls podName:947050c5-395e-4330-8557-271ad164c107 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:23.434381253 +0000 UTC m=+51.136882438 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls") pod "image-registry-94464db7-rhhq7" (UID: "947050c5-395e-4330-8557-271ad164c107") : secret "image-registry-tls" not found Jul 09 17:40:15.535456 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:15.535400 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:15.535638 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:15.535561 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 17:40:15.535638 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:15.535631 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls podName:40f09dde-c76e-410b-98cd-cdb1de2858d4 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:23.535611627 +0000 UTC m=+51.238112795 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls") pod "dns-default-6dtwp" (UID: "40f09dde-c76e-410b-98cd-cdb1de2858d4") : secret "dns-default-metrics-tls" not found Jul 09 17:40:19.429664 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:19.429626 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:19.533595 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:19.533563 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:21.444459 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:21.444373 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:21.509924 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:21.509886 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:22.249107 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:22.249072 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:22.323544 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:22.323509 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:23.495952 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:23.495906 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:23.496368 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:23.496063 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 17:40:23.496368 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:23.496083 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94464db7-rhhq7: secret "image-registry-tls" not found Jul 09 17:40:23.496368 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:23.496147 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls podName:947050c5-395e-4330-8557-271ad164c107 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:39.496129287 +0000 UTC m=+67.198630455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls") pod "image-registry-94464db7-rhhq7" (UID: "947050c5-395e-4330-8557-271ad164c107") : secret "image-registry-tls" not found Jul 09 17:40:23.597213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:23.597178 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:23.597370 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:23.597334 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 17:40:23.597414 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:23.597408 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls podName:40f09dde-c76e-410b-98cd-cdb1de2858d4 nodeName:}" failed. No retries permitted until 2026-07-09 17:40:39.597391556 +0000 UTC m=+67.299892725 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls") pod "dns-default-6dtwp" (UID: "40f09dde-c76e-410b-98cd-cdb1de2858d4") : secret "dns-default-metrics-tls" not found Jul 09 17:40:23.865779 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:23.865688 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:23.938907 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:23.938865 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:24.655552 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:24.655511 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:26.153532 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:26.153496 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:26.461747 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:26.461712 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:26.530359 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:26.530324 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:33.968332 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:33.968304 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xxd4" Jul 09 17:40:34.174677 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:34.174625 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:34.272797 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:34.272722 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:38.514913 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.514861 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:40:38.517399 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.517379 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 17:40:38.525228 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:38.525203 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 17:40:38.525359 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:38.525274 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs podName:86dbd8b7-a392-48da-901a-a23b1fabec38 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:42.525255073 +0000 UTC m=+130.227756240 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs") pod "network-metrics-daemon-7vlf2" (UID: "86dbd8b7-a392-48da-901a-a23b1fabec38") : secret "metrics-daemon-secret" not found Jul 09 17:40:38.616103 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.616064 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:38.616103 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.616105 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:38.619605 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.619583 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 17:40:38.619704 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.619583 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 17:40:38.629736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.629708 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 17:40:38.631162 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.631137 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7-original-pull-secret\") pod \"global-pull-secret-syncer-7wrgh\" (UID: \"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7\") " pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:38.639943 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.639914 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qdgs\" (UniqueName: \"kubernetes.io/projected/3854aae9-5583-4345-a96c-9f3f2f90cbb3-kube-api-access-5qdgs\") pod \"network-check-target-rdfxs\" (UID: \"3854aae9-5583-4345-a96c-9f3f2f90cbb3\") " pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:38.718444 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.718390 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-7wrgh" Jul 09 17:40:38.727802 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.727768 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9rs28\"" Jul 09 17:40:38.735951 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.735912 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:38.885700 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.885666 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-7wrgh"] Jul 09 17:40:38.891265 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:40:38.891232 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ff51e9_bbc2_419e_84f4_44b56c5ae8d7.slice/crio-08ff8a80f34738086e5751f0b89fb3b7a3b5fcbdaa7f4f857ddce690a1e23b60 WatchSource:0}: Error finding container 08ff8a80f34738086e5751f0b89fb3b7a3b5fcbdaa7f4f857ddce690a1e23b60: Status 404 returned error can't find the container with id 08ff8a80f34738086e5751f0b89fb3b7a3b5fcbdaa7f4f857ddce690a1e23b60 Jul 09 17:40:38.911271 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:38.911236 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rdfxs"] Jul 09 17:40:38.914053 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:40:38.914019 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3854aae9_5583_4345_a96c_9f3f2f90cbb3.slice/crio-c0c8e73771eec47dc2d6b016d67a57154e85e4733989dbda0f481459c706686f WatchSource:0}: Error finding container c0c8e73771eec47dc2d6b016d67a57154e85e4733989dbda0f481459c706686f: Status 404 returned error can't find the container with id c0c8e73771eec47dc2d6b016d67a57154e85e4733989dbda0f481459c706686f Jul 09 17:40:39.050407 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:39.050307 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rdfxs" event={"ID":"3854aae9-5583-4345-a96c-9f3f2f90cbb3","Type":"ContainerStarted","Data":"c0c8e73771eec47dc2d6b016d67a57154e85e4733989dbda0f481459c706686f"} Jul 09 17:40:39.051487 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:39.051452 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7wrgh" event={"ID":"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7","Type":"ContainerStarted","Data":"08ff8a80f34738086e5751f0b89fb3b7a3b5fcbdaa7f4f857ddce690a1e23b60"} Jul 09 17:40:39.532025 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:39.531943 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:40:39.532536 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:39.532108 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 17:40:39.532536 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:39.532130 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94464db7-rhhq7: secret "image-registry-tls" not found Jul 09 17:40:39.532536 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:39.532191 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls podName:947050c5-395e-4330-8557-271ad164c107 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.532173736 +0000 UTC m=+99.234674928 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls") pod "image-registry-94464db7-rhhq7" (UID: "947050c5-395e-4330-8557-271ad164c107") : secret "image-registry-tls" not found Jul 09 17:40:39.633205 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:39.633162 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:40:39.633382 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:39.633359 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 17:40:39.633483 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:40:39.633470 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls podName:40f09dde-c76e-410b-98cd-cdb1de2858d4 nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.633442441 +0000 UTC m=+99.335943626 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls") pod "dns-default-6dtwp" (UID: "40f09dde-c76e-410b-98cd-cdb1de2858d4") : secret "dns-default-metrics-tls" not found Jul 09 17:40:44.069093 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:44.068975 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rdfxs" event={"ID":"3854aae9-5583-4345-a96c-9f3f2f90cbb3","Type":"ContainerStarted","Data":"27c4e8b7af3b1c4e76611292bd56ba8bb12de058982b29bb7c8173976ead900b"} Jul 09 17:40:44.069610 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:44.069135 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:40:44.070516 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:44.070482 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-7wrgh" event={"ID":"c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7","Type":"ContainerStarted","Data":"3b8a25bde04b56c6b7477b23293bdd985c496283db4f16e698b181327bf87fce"} Jul 09 17:40:44.091030 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:44.090945 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rdfxs" podStartSLOduration=66.303900399 podStartE2EDuration="1m11.090928182s" totalImagesPullingTime="4.787027783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:44.090816247 +0000 UTC m=+71.793317437" watchObservedRunningTime="2026-07-09 17:40:44.090928182 +0000 UTC m=+71.793429364" Jul 09 17:40:46.009162 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:46.009113 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:46.113040 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:46.113003 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:47.305018 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:47.304964 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:40:47.489809 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:40:47.489770 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:04.098185 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.098011 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-7wrgh" podStartSLOduration=86.284106933 podStartE2EDuration="1m31.097993454s" totalImagesPullingTime="4.813886521s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:40:44.108578247 +0000 UTC m=+71.811079437" watchObservedRunningTime="2026-07-09 17:41:04.097993454 +0000 UTC m=+91.800494644" Jul 09 17:41:04.098766 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.098272 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck"] Jul 09 17:41:04.101256 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.101233 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck" Jul 09 17:41:04.104566 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.104548 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7s54x\"" Jul 09 17:41:04.110876 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.110850 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck"] Jul 09 17:41:04.146312 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.146279 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb"] Jul 09 17:41:04.148848 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.148831 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.151588 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.151561 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-2r4hm\"" Jul 09 17:41:04.151809 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.151792 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 17:41:04.151939 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.151918 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 17:41:04.152070 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.152054 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 17:41:04.152305 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.152283 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 17:41:04.153461 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.153442 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6"] Jul 09 17:41:04.155576 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.155555 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:04.164235 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.164212 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 17:41:04.165328 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.165311 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 17:41:04.165438 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.165406 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-47rqz\"" Jul 09 17:41:04.165933 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.165916 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 17:41:04.167799 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.167777 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb"] Jul 09 17:41:04.182630 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.182593 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6"] Jul 09 17:41:04.219743 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.219698 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxcxs\" (UniqueName: \"kubernetes.io/projected/53a6095e-30c2-416a-8bd3-136362053252-kube-api-access-sxcxs\") pod \"network-check-source-869dcfcdcd-pbjck\" (UID: \"53a6095e-30c2-416a-8bd3-136362053252\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck" Jul 09 17:41:04.219937 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.219758 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfm6\" (UniqueName: \"kubernetes.io/projected/62d18c03-835d-48a0-86e6-7a1f9f0d9882-kube-api-access-nlfm6\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.219937 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.219780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvwm4\" (UniqueName: \"kubernetes.io/projected/e7430757-5c42-4538-943d-0e1493ad00fa-kube-api-access-jvwm4\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:04.219937 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.219886 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62d18c03-835d-48a0-86e6-7a1f9f0d9882-serving-cert\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.219937 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.219920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:04.220070 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.219978 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d18c03-835d-48a0-86e6-7a1f9f0d9882-config\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.311384 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.311352 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:04.321044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.321012 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d18c03-835d-48a0-86e6-7a1f9f0d9882-config\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.321368 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.321060 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxcxs\" (UniqueName: \"kubernetes.io/projected/53a6095e-30c2-416a-8bd3-136362053252-kube-api-access-sxcxs\") pod \"network-check-source-869dcfcdcd-pbjck\" (UID: \"53a6095e-30c2-416a-8bd3-136362053252\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck" Jul 09 17:41:04.321368 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.321164 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfm6\" (UniqueName: \"kubernetes.io/projected/62d18c03-835d-48a0-86e6-7a1f9f0d9882-kube-api-access-nlfm6\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.321368 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.321182 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvwm4\" (UniqueName: \"kubernetes.io/projected/e7430757-5c42-4538-943d-0e1493ad00fa-kube-api-access-jvwm4\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:04.321368 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.321212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62d18c03-835d-48a0-86e6-7a1f9f0d9882-serving-cert\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.321607 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.321367 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:04.321607 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:04.321534 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 17:41:04.321705 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:04.321629 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls podName:e7430757-5c42-4538-943d-0e1493ad00fa nodeName:}" failed. No retries permitted until 2026-07-09 17:41:04.821606726 +0000 UTC m=+92.524107907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls") pod "cluster-samples-operator-65db858894-w8db6" (UID: "e7430757-5c42-4538-943d-0e1493ad00fa") : secret "samples-operator-tls" not found Jul 09 17:41:04.321763 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.321700 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/62d18c03-835d-48a0-86e6-7a1f9f0d9882-config\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.323592 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.323565 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/62d18c03-835d-48a0-86e6-7a1f9f0d9882-serving-cert\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.330327 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.330297 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfm6\" (UniqueName: \"kubernetes.io/projected/62d18c03-835d-48a0-86e6-7a1f9f0d9882-kube-api-access-nlfm6\") pod \"service-ca-operator-7695fc6c8c-cvvnb\" (UID: \"62d18c03-835d-48a0-86e6-7a1f9f0d9882\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.330670 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.330649 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvwm4\" (UniqueName: \"kubernetes.io/projected/e7430757-5c42-4538-943d-0e1493ad00fa-kube-api-access-jvwm4\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:04.330722 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.330648 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxcxs\" (UniqueName: \"kubernetes.io/projected/53a6095e-30c2-416a-8bd3-136362053252-kube-api-access-sxcxs\") pod \"network-check-source-869dcfcdcd-pbjck\" (UID: \"53a6095e-30c2-416a-8bd3-136362053252\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck" Jul 09 17:41:04.410570 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.410464 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck" Jul 09 17:41:04.412855 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.412824 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:04.459118 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.459080 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" Jul 09 17:41:04.546412 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.546380 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck"] Jul 09 17:41:04.550225 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:41:04.550196 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53a6095e_30c2_416a_8bd3_136362053252.slice/crio-caff20ba0280ea5c6d5872bf90ef2dd4d0ab2a2c3fae121a84647d711987160a WatchSource:0}: Error finding container caff20ba0280ea5c6d5872bf90ef2dd4d0ab2a2c3fae121a84647d711987160a: Status 404 returned error can't find the container with id caff20ba0280ea5c6d5872bf90ef2dd4d0ab2a2c3fae121a84647d711987160a Jul 09 17:41:04.592747 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.592718 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb"] Jul 09 17:41:04.595819 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:41:04.595792 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62d18c03_835d_48a0_86e6_7a1f9f0d9882.slice/crio-f052be1a876fae18683fc915389a7348399ac4942671752ad1bfc4c4a0b9cafe WatchSource:0}: Error finding container f052be1a876fae18683fc915389a7348399ac4942671752ad1bfc4c4a0b9cafe: Status 404 returned error can't find the container with id f052be1a876fae18683fc915389a7348399ac4942671752ad1bfc4c4a0b9cafe Jul 09 17:41:04.826203 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:04.826158 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:04.826399 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:04.826326 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 17:41:04.826399 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:04.826398 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls podName:e7430757-5c42-4538-943d-0e1493ad00fa nodeName:}" failed. No retries permitted until 2026-07-09 17:41:05.826381319 +0000 UTC m=+93.528882492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls") pod "cluster-samples-operator-65db858894-w8db6" (UID: "e7430757-5c42-4538-943d-0e1493ad00fa") : secret "samples-operator-tls" not found Jul 09 17:41:05.125578 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:05.125444 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" event={"ID":"62d18c03-835d-48a0-86e6-7a1f9f0d9882","Type":"ContainerStarted","Data":"f052be1a876fae18683fc915389a7348399ac4942671752ad1bfc4c4a0b9cafe"} Jul 09 17:41:05.127312 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:05.127274 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck" event={"ID":"53a6095e-30c2-416a-8bd3-136362053252","Type":"ContainerStarted","Data":"6dc3d1beaea654596a262cad880091d4d8aa5e5869e13eab6909e6c5bd5dca37"} Jul 09 17:41:05.127487 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:05.127333 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck" event={"ID":"53a6095e-30c2-416a-8bd3-136362053252","Type":"ContainerStarted","Data":"caff20ba0280ea5c6d5872bf90ef2dd4d0ab2a2c3fae121a84647d711987160a"} Jul 09 17:41:05.144366 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:05.144301 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-pbjck" podStartSLOduration=1.144272906 podStartE2EDuration="1.144272906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:05.143352598 +0000 UTC m=+92.845853788" watchObservedRunningTime="2026-07-09 17:41:05.144272906 +0000 UTC m=+92.846774100" Jul 09 17:41:05.834252 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:05.834218 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:05.834466 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:05.834401 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 17:41:05.834531 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:05.834503 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls podName:e7430757-5c42-4538-943d-0e1493ad00fa nodeName:}" failed. No retries permitted until 2026-07-09 17:41:07.834479677 +0000 UTC m=+95.536980846 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls") pod "cluster-samples-operator-65db858894-w8db6" (UID: "e7430757-5c42-4538-943d-0e1493ad00fa") : secret "samples-operator-tls" not found Jul 09 17:41:05.921567 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:05.921533 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:06.021826 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:06.021792 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:07.134398 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:07.134361 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" event={"ID":"62d18c03-835d-48a0-86e6-7a1f9f0d9882","Type":"ContainerStarted","Data":"e4a7d9acb79bb5a8a3d13d1a843bd5e9c2071270001fca498244a2078f10b3f8"} Jul 09 17:41:07.152107 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:07.152055 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" podStartSLOduration=1.2711623730000001 podStartE2EDuration="3.152038792s" totalImagesPullingTime="1.880876419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:07.151073919 +0000 UTC m=+94.853575108" watchObservedRunningTime="2026-07-09 17:41:07.152038792 +0000 UTC m=+94.854539982" Jul 09 17:41:07.719646 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:07.719600 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:07.818142 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:07.818111 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:07.851483 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:07.851411 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:07.851682 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:07.851536 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 17:41:07.851682 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:07.851608 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls podName:e7430757-5c42-4538-943d-0e1493ad00fa nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.851591194 +0000 UTC m=+99.554092362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls") pod "cluster-samples-operator-65db858894-w8db6" (UID: "e7430757-5c42-4538-943d-0e1493ad00fa") : secret "samples-operator-tls" not found Jul 09 17:41:08.000541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.000458 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xj9gk"] Jul 09 17:41:08.002586 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.002569 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.005675 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.005652 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 17:41:08.005675 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.005674 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 17:41:08.006756 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.006735 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 17:41:08.006844 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.006793 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 17:41:08.006910 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.006802 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n86q2\"" Jul 09 17:41:08.028180 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.028146 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xj9gk"] Jul 09 17:41:08.070779 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.070741 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:08.144203 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.144170 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:08.153330 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.153292 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7fc605-af53-4640-bda0-d873c2a53eeb-crio-socket\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.153484 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.153337 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk2ht\" (UniqueName: \"kubernetes.io/projected/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-api-access-mk2ht\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.153484 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.153450 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.153566 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.153507 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7fc605-af53-4640-bda0-d873c2a53eeb-data-volume\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.153600 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.153561 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.244261 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.244224 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:08.254005 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.253921 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7fc605-af53-4640-bda0-d873c2a53eeb-crio-socket\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.254005 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.253960 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk2ht\" (UniqueName: \"kubernetes.io/projected/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-api-access-mk2ht\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.254198 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.254028 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.254198 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.254070 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7fc605-af53-4640-bda0-d873c2a53eeb-data-volume\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.254198 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.254103 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.254198 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:08.254149 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 17:41:08.254473 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:08.254226 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:08.754203831 +0000 UTC m=+96.456705018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : secret "insights-runtime-extractor-tls" not found Jul 09 17:41:08.254473 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:08.254244 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 17:41:08.254473 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.254070 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ab7fc605-af53-4640-bda0-d873c2a53eeb-crio-socket\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.254473 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:08.254298 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:08.754282722 +0000 UTC m=+96.456783890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : configmap "kube-rbac-proxy" not found Jul 09 17:41:08.254617 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.254501 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ab7fc605-af53-4640-bda0-d873c2a53eeb-data-volume\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.270611 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.270573 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk2ht\" (UniqueName: \"kubernetes.io/projected/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-api-access-mk2ht\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.348173 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.348130 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:08.445136 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.445100 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:08.583972 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.583887 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:08.759398 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.759357 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.759612 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:08.759498 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:08.759612 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:08.759511 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 17:41:08.759612 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:08.759578 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:09.759562233 +0000 UTC m=+97.462063423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : configmap "kube-rbac-proxy" not found Jul 09 17:41:08.759774 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:08.759636 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 17:41:08.759774 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:08.759695 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:09.759678773 +0000 UTC m=+97.462179944 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : secret "insights-runtime-extractor-tls" not found Jul 09 17:41:09.767475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:09.767414 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:09.767978 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:09.767500 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:09.767978 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:09.767529 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 17:41:09.767978 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:09.767598 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.767583117 +0000 UTC m=+99.470084285 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : secret "insights-runtime-extractor-tls" not found Jul 09 17:41:09.767978 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:09.767597 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 17:41:09.767978 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:09.767628 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:11.767620184 +0000 UTC m=+99.470121352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : configmap "kube-rbac-proxy" not found Jul 09 17:41:10.000313 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.000287 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzm26_1c43ac19-a9e4-47c7-a4ca-a5d758033df8/dns-node-resolver/0.log" Jul 09 17:41:10.025791 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.025711 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ljq8x"] Jul 09 17:41:10.027977 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.027959 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.030089 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.030064 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 09 17:41:10.030254 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.030118 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 09 17:41:10.030254 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.030144 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 09 17:41:10.030796 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.030781 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-z2gq6\"" Jul 09 17:41:10.030866 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.030808 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 09 17:41:10.038264 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.038236 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ljq8x"] Jul 09 17:41:10.065915 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.065884 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:10.140126 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.140087 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:10.170031 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.170003 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-signing-cabundle\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.170216 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.170049 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-signing-key\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.170216 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.170127 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbpqb\" (UniqueName: \"kubernetes.io/projected/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-kube-api-access-xbpqb\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.271523 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.271479 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-signing-key\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.271740 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.271541 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbpqb\" (UniqueName: \"kubernetes.io/projected/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-kube-api-access-xbpqb\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.271740 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.271604 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-signing-cabundle\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.272242 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.272223 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-signing-cabundle\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.274096 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.274071 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-signing-key\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.286387 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.286279 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbpqb\" (UniqueName: \"kubernetes.io/projected/43afa3af-b72e-4ef6-b69a-fe7cd8982e92-kube-api-access-xbpqb\") pod \"service-ca-79f68b9f4f-ljq8x\" (UID: \"43afa3af-b72e-4ef6-b69a-fe7cd8982e92\") " pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.338095 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.338046 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" Jul 09 17:41:10.446114 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.446059 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:10.471018 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.470984 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-ljq8x"] Jul 09 17:41:10.474552 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:41:10.474526 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43afa3af_b72e_4ef6_b69a_fe7cd8982e92.slice/crio-f72791af6100cf638aca529af258527128e1ec224d74f0db8d52dba69c3481c1 WatchSource:0}: Error finding container f72791af6100cf638aca529af258527128e1ec224d74f0db8d52dba69c3481c1: Status 404 returned error can't find the container with id f72791af6100cf638aca529af258527128e1ec224d74f0db8d52dba69c3481c1 Jul 09 17:41:10.550222 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.550136 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:10.795764 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:10.795739 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4b8tl_a0114f7e-8bb8-40ce-a472-15872a204f8a/node-ca/0.log" Jul 09 17:41:11.025990 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.025954 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:11.146483 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.146448 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" event={"ID":"43afa3af-b72e-4ef6-b69a-fe7cd8982e92","Type":"ContainerStarted","Data":"4357384a4a7fb69b6f30e0b8e49d2d14652a0fc13b7b53dd2bcf5efe64b8174e"} Jul 09 17:41:11.146483 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.146501 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" event={"ID":"43afa3af-b72e-4ef6-b69a-fe7cd8982e92","Type":"ContainerStarted","Data":"f72791af6100cf638aca529af258527128e1ec224d74f0db8d52dba69c3481c1"} Jul 09 17:41:11.175561 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.175502 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-ljq8x" podStartSLOduration=1.175481567 podStartE2EDuration="1.175481567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:11.173937457 +0000 UTC m=+98.876438646" watchObservedRunningTime="2026-07-09 17:41:11.175481567 +0000 UTC m=+98.877982835" Jul 09 17:41:11.582872 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.582825 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") pod \"image-registry-94464db7-rhhq7\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:41:11.583059 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.582983 2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 17:41:11.583059 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.582999 2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-94464db7-rhhq7: secret "image-registry-tls" not found Jul 09 17:41:11.583059 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.583058 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls podName:947050c5-395e-4330-8557-271ad164c107 nodeName:}" failed. No retries permitted until 2026-07-09 17:42:15.58304141 +0000 UTC m=+163.285542585 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls") pod "image-registry-94464db7-rhhq7" (UID: "947050c5-395e-4330-8557-271ad164c107") : secret "image-registry-tls" not found Jul 09 17:41:11.683865 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.683802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:41:11.684089 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.683974 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 17:41:11.684089 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.684050 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls podName:40f09dde-c76e-410b-98cd-cdb1de2858d4 nodeName:}" failed. No retries permitted until 2026-07-09 17:42:15.684033249 +0000 UTC m=+163.386534425 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls") pod "dns-default-6dtwp" (UID: "40f09dde-c76e-410b-98cd-cdb1de2858d4") : secret "dns-default-metrics-tls" not found Jul 09 17:41:11.784387 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.784349 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:11.784585 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.784494 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:11.784585 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.784520 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 17:41:11.784667 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.784589 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:15.784572091 +0000 UTC m=+103.487073271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : secret "insights-runtime-extractor-tls" not found Jul 09 17:41:11.784667 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.784606 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 17:41:11.784667 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.784658 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:15.784645426 +0000 UTC m=+103.487146594 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : configmap "kube-rbac-proxy" not found Jul 09 17:41:11.885595 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.885493 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:11.885985 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.885682 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 17:41:11.885985 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:11.885774 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls podName:e7430757-5c42-4538-943d-0e1493ad00fa nodeName:}" failed. No retries permitted until 2026-07-09 17:41:19.885751366 +0000 UTC m=+107.588252537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls") pod "cluster-samples-operator-65db858894-w8db6" (UID: "e7430757-5c42-4538-943d-0e1493ad00fa") : secret "samples-operator-tls" not found Jul 09 17:41:11.996248 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:11.996218 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mcntj_aca88050-a7a2-4388-b3bd-20c66b173963/serve-healthcheck-canary/0.log" Jul 09 17:41:12.104604 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:12.104569 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:12.298451 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:12.298395 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:12.398130 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:12.398082 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:12.496960 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:12.496909 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:12.599588 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:12.599491 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:12.721789 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:12.721758 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:12.804060 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:12.804024 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:14.028775 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:14.028742 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:14.204851 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:14.204819 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:14.400302 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:14.400202 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:14.501238 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:14.501206 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:14.601326 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:14.601287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:14.804468 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:14.804432 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:15.076874 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:15.076787 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rdfxs" Jul 09 17:41:15.529028 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:15.528993 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:15.799971 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:15.799884 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:15.820489 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:15.820452 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:15.820650 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:15.820542 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:15.820650 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:15.820614 2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 17:41:15.820650 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:15.820619 2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 17:41:15.820788 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:15.820672 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:23.820657382 +0000 UTC m=+111.523158549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : configmap "kube-rbac-proxy" not found Jul 09 17:41:15.820788 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:15.820701 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls podName:ab7fc605-af53-4640-bda0-d873c2a53eeb nodeName:}" failed. No retries permitted until 2026-07-09 17:41:23.820681131 +0000 UTC m=+111.523182316 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-xj9gk" (UID: "ab7fc605-af53-4640-bda0-d873c2a53eeb") : secret "insights-runtime-extractor-tls" not found Jul 09 17:41:15.998119 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:15.998088 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:16.207729 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:16.207691 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:16.405357 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:16.405323 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:16.506955 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:16.506866 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:16.606365 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:16.606332 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:16.793385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:16.793295 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:17.796691 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:17.796639 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:18.197243 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:18.197222 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:18.316573 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:18.316532 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:18.416544 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:18.416509 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:18.596687 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:18.596592 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:19.026274 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:19.026193 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:19.793044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:19.793010 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:19.960433 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:19.960383 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:19.962880 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:19.962842 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e7430757-5c42-4538-943d-0e1493ad00fa-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-w8db6\" (UID: \"e7430757-5c42-4538-943d-0e1493ad00fa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:20.004860 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:20.004824 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:20.065329 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:20.065244 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" Jul 09 17:41:20.198615 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:20.198580 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6"] Jul 09 17:41:20.394801 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:20.394679 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:20.602123 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:20.602090 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:21.182576 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:21.182540 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" event={"ID":"e7430757-5c42-4538-943d-0e1493ad00fa","Type":"ContainerStarted","Data":"6fe95adf95d0cc9f2f4bbdf4e788ec1d5b3213377c9fb623bb6e4e584aefc9cb"} Jul 09 17:41:21.198734 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:21.198703 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:21.299000 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:21.298960 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:21.657550 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:21.657509 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:21.800819 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:21.800765 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:22.029924 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:22.029891 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:22.187100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:22.186995 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" event={"ID":"e7430757-5c42-4538-943d-0e1493ad00fa","Type":"ContainerStarted","Data":"e2ee04113f2279ce305a506e4d891b82ef2bf7b9b5d95c0906b93be5f15e32b6"} Jul 09 17:41:22.187100 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:22.187065 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" event={"ID":"e7430757-5c42-4538-943d-0e1493ad00fa","Type":"ContainerStarted","Data":"9e253e97a12ad6337ffa9a77187de9565245a04d235dd6368a659f8fe5fb38f1"} Jul 09 17:41:22.206026 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:22.205964 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-w8db6" podStartSLOduration=16.553122368 podStartE2EDuration="18.205946977s" totalImagesPullingTime="1.652824609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:22.204670865 +0000 UTC m=+109.907172057" watchObservedRunningTime="2026-07-09 17:41:22.205946977 +0000 UTC m=+109.908448166" Jul 09 17:41:22.209962 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:22.209930 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:22.300156 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:22.300125 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:22.599760 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:22.599724 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:22.929122 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:22.929028 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:23.033722 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:23.033684 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:23.204254 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:23.204225 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:23.898073 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:23.898036 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:23.898279 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:23.898117 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:23.898656 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:23.898638 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ab7fc605-af53-4640-bda0-d873c2a53eeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:23.900660 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:23.900641 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ab7fc605-af53-4640-bda0-d873c2a53eeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xj9gk\" (UID: \"ab7fc605-af53-4640-bda0-d873c2a53eeb\") " pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:23.911648 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:23.911610 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xj9gk" Jul 09 17:41:24.043285 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:24.043246 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xj9gk"] Jul 09 17:41:24.046817 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:41:24.046785 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7fc605_af53_4640_bda0_d873c2a53eeb.slice/crio-27d23508e24da8fa5665cad0cc2c2f9fa2ec24773d81d473fdaee7f7dd9a70a1 WatchSource:0}: Error finding container 27d23508e24da8fa5665cad0cc2c2f9fa2ec24773d81d473fdaee7f7dd9a70a1: Status 404 returned error can't find the container with id 27d23508e24da8fa5665cad0cc2c2f9fa2ec24773d81d473fdaee7f7dd9a70a1 Jul 09 17:41:24.193941 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:24.193854 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xj9gk" event={"ID":"ab7fc605-af53-4640-bda0-d873c2a53eeb","Type":"ContainerStarted","Data":"27d23508e24da8fa5665cad0cc2c2f9fa2ec24773d81d473fdaee7f7dd9a70a1"} Jul 09 17:41:28.209550 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:28.209512 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xj9gk" event={"ID":"ab7fc605-af53-4640-bda0-d873c2a53eeb","Type":"ContainerStarted","Data":"402490c064088f26c2dc5a60b916824e715f9d831093c4c1d0fc0078e65c852d"} Jul 09 17:41:29.219800 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.219753 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xj9gk" event={"ID":"ab7fc605-af53-4640-bda0-d873c2a53eeb","Type":"ContainerStarted","Data":"6d69143c1c2f38cfb266216df4677fc08191afffc366cd04210eec22421212e2"} Jul 09 17:41:29.356495 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.356460 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6"] Jul 09 17:41:29.358924 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.358905 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.361767 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.361738 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 09 17:41:29.362061 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.362041 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 09 17:41:29.362172 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.362153 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 09 17:41:29.363196 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.363173 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 09 17:41:29.363318 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.363253 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 09 17:41:29.363318 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.363302 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 09 17:41:29.363457 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.363336 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 09 17:41:29.387748 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.387711 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6"] Jul 09 17:41:29.444566 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.444528 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.444758 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.444643 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89dc92fc-2ec1-4a5e-9414-103b405410c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.444758 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.444698 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-hub\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.444758 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.444727 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.444932 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.444790 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-ca\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.444932 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.444831 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4qcf\" (UniqueName: \"kubernetes.io/projected/89dc92fc-2ec1-4a5e-9414-103b405410c1-kube-api-access-m4qcf\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.467054 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.467014 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:29.545888 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.545793 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89dc92fc-2ec1-4a5e-9414-103b405410c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.545888 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.545870 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-hub\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.546122 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.545903 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.546122 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.545930 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-ca\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.546122 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.545952 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4qcf\" (UniqueName: \"kubernetes.io/projected/89dc92fc-2ec1-4a5e-9414-103b405410c1-kube-api-access-m4qcf\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.546260 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.546190 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.546724 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.546672 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/89dc92fc-2ec1-4a5e-9414-103b405410c1-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.548985 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.548930 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-ca\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.549113 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.549031 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.549257 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.549239 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-hub\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.549531 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.549513 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/89dc92fc-2ec1-4a5e-9414-103b405410c1-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.580092 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.580054 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4qcf\" (UniqueName: \"kubernetes.io/projected/89dc92fc-2ec1-4a5e-9414-103b405410c1-kube-api-access-m4qcf\") pod \"cluster-proxy-proxy-agent-d96c6455c-7s8q6\" (UID: \"89dc92fc-2ec1-4a5e-9414-103b405410c1\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.625987 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.625934 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:29.680098 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.680051 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" Jul 09 17:41:29.717041 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.717005 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:29.818573 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.818488 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:29.837668 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:29.837567 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6"] Jul 09 17:41:29.844036 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:41:29.843995 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89dc92fc_2ec1_4a5e_9414_103b405410c1.slice/crio-bcc71f33ab51cb46678a72da426a36f2128f8a02be5b6a2b0e18e1ac06c2d314 WatchSource:0}: Error finding container bcc71f33ab51cb46678a72da426a36f2128f8a02be5b6a2b0e18e1ac06c2d314: Status 404 returned error can't find the container with id bcc71f33ab51cb46678a72da426a36f2128f8a02be5b6a2b0e18e1ac06c2d314 Jul 09 17:41:30.224773 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:30.224736 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" event={"ID":"89dc92fc-2ec1-4a5e-9414-103b405410c1","Type":"ContainerStarted","Data":"bcc71f33ab51cb46678a72da426a36f2128f8a02be5b6a2b0e18e1ac06c2d314"} Jul 09 17:41:30.785598 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:30.785496 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:31.147803 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:31.147761 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:31.232196 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:31.232159 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xj9gk" event={"ID":"ab7fc605-af53-4640-bda0-d873c2a53eeb","Type":"ContainerStarted","Data":"bc5d4d041684e5fa4a50946bbb575da3cbc4aacb57f183580931dc25bef68dc5"} Jul 09 17:41:31.246373 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:31.246286 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:31.254358 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:31.254296 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xj9gk" podStartSLOduration=17.429889392 podStartE2EDuration="24.254278487s" totalImagesPullingTime="6.824389095s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:41:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:31.253316758 +0000 UTC m=+118.955817949" watchObservedRunningTime="2026-07-09 17:41:31.254278487 +0000 UTC m=+118.956779678" Jul 09 17:41:32.428843 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:32.428799 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:33.229876 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:33.229836 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:33.315038 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:33.315000 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:33.801179 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:33.801142 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:34.244511 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:34.244464 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" event={"ID":"89dc92fc-2ec1-4a5e-9414-103b405410c1","Type":"ContainerStarted","Data":"6fac9d33e4c50e62beafd77d244c8c6bba171763910045ac39b0c9e4ef17ab0c"} Jul 09 17:41:34.244690 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:34.244526 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" event={"ID":"89dc92fc-2ec1-4a5e-9414-103b405410c1","Type":"ContainerStarted","Data":"989739a56b671ec905da7a5c2cea368a2312e43dcb5dd5dd6f837f8bba21be50"} Jul 09 17:41:34.244690 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:34.244543 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" event={"ID":"89dc92fc-2ec1-4a5e-9414-103b405410c1","Type":"ContainerStarted","Data":"40ae5c7c995b52d0bbcee619a08a2b2b95a2818bb0c609c37762740e3402b124"} Jul 09 17:41:34.249643 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:34.249616 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:34.264270 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:34.264219 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d96c6455c-7s8q6" podStartSLOduration=1.137048748 podStartE2EDuration="5.264202397s" totalImagesPullingTime="4.127153649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:34.263326585 +0000 UTC m=+121.965827775" watchObservedRunningTime="2026-07-09 17:41:34.264202397 +0000 UTC m=+121.966703638" Jul 09 17:41:34.522865 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:34.522831 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:34.584251 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:34.584218 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:34.641975 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:34.641946 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:35.778977 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:35.778945 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:37.371696 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.371660 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:37.628530 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.628400 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:37.745534 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.745500 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-dmdv5"] Jul 09 17:41:37.748773 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.748748 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.750941 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.750905 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9krtl\"" Jul 09 17:41:37.751108 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.751014 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 17:41:37.751108 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.751014 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 17:41:37.751541 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.751518 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 17:41:37.751667 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.751577 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 17:41:37.751667 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.751640 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 17:41:37.751778 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.751754 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 17:41:37.785197 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.785168 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:37.815795 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.815756 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-root\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.815795 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.815795 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.816025 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.815836 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-tls\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.816025 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.815857 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.816025 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.815978 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-textfile\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.816167 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.816034 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08596c60-5055-4866-a866-18b83867ba3c-metrics-client-ca\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.816167 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.816146 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-sys\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.816250 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.816181 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmh4\" (UniqueName: \"kubernetes.io/projected/08596c60-5055-4866-a866-18b83867ba3c-kube-api-access-8nmh4\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.816290 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.816242 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-wtmp\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.879762 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.879671 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:37.916843 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.916802 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-textfile\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917027 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.916852 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08596c60-5055-4866-a866-18b83867ba3c-metrics-client-ca\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917027 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.916897 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-sys\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917027 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.916915 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmh4\" (UniqueName: \"kubernetes.io/projected/08596c60-5055-4866-a866-18b83867ba3c-kube-api-access-8nmh4\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917027 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.916945 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-wtmp\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917027 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.916976 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-root\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917027 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.916991 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-sys\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917027 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.916999 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.917045 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-tls\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.917076 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.917176 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-textfile\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.917209 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-root\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917612 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.917564 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08596c60-5055-4866-a866-18b83867ba3c-metrics-client-ca\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917612 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.917599 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-wtmp\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.917720 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.917653 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.919596 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.919572 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-tls\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.919713 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.919656 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08596c60-5055-4866-a866-18b83867ba3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.927245 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.927211 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmh4\" (UniqueName: \"kubernetes.io/projected/08596c60-5055-4866-a866-18b83867ba3c-kube-api-access-8nmh4\") pod \"node-exporter-dmdv5\" (UID: \"08596c60-5055-4866-a866-18b83867ba3c\") " pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:37.980322 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:37.980287 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:38.058565 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:38.058531 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-dmdv5" Jul 09 17:41:38.071548 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:41:38.071512 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08596c60_5055_4866_a866_18b83867ba3c.slice/crio-84f9287b1f3da213d2b2184b72ea3f4dcd3fec6dce35adf0849c03a415db856d WatchSource:0}: Error finding container 84f9287b1f3da213d2b2184b72ea3f4dcd3fec6dce35adf0849c03a415db856d: Status 404 returned error can't find the container with id 84f9287b1f3da213d2b2184b72ea3f4dcd3fec6dce35adf0849c03a415db856d Jul 09 17:41:38.083116 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:38.083087 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:38.189128 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:38.189030 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:38.260401 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:38.260350 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dmdv5" event={"ID":"08596c60-5055-4866-a866-18b83867ba3c","Type":"ContainerStarted","Data":"84f9287b1f3da213d2b2184b72ea3f4dcd3fec6dce35adf0849c03a415db856d"} Jul 09 17:41:38.630323 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:38.630275 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:38.708481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:38.708414 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:39.158994 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:39.158952 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:40.267630 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:40.267594 2560 generic.go:356] "Generic (PLEG): container finished" podID="08596c60-5055-4866-a866-18b83867ba3c" containerID="f640f6c094d48ba518bea1c2fe1f636b6e4f211b0277b24aa84e9fecfb6c8d5d" exitCode=0 Jul 09 17:41:40.267630 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:40.267634 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dmdv5" event={"ID":"08596c60-5055-4866-a866-18b83867ba3c","Type":"ContainerDied","Data":"f640f6c094d48ba518bea1c2fe1f636b6e4f211b0277b24aa84e9fecfb6c8d5d"} Jul 09 17:41:40.631502 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:40.631394 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:40.695713 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:40.695680 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:41.043150 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:41.043112 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:41.185343 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:41.185308 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:41.272879 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:41.272843 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dmdv5" event={"ID":"08596c60-5055-4866-a866-18b83867ba3c","Type":"ContainerStarted","Data":"f5055cd059cedbfef1c3b25fe353078ea547e36cd9cbc1624d15f02183d34017"} Jul 09 17:41:41.273347 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:41.272908 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-dmdv5" event={"ID":"08596c60-5055-4866-a866-18b83867ba3c","Type":"ContainerStarted","Data":"dcb6f64078515e978cbfbc3015a4555247bf416809fa3de05dc3a6f4ca863687"} Jul 09 17:41:41.292068 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:41.292013 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-dmdv5" podStartSLOduration=2.9956442020000003 podStartE2EDuration="4.291995195s" totalImagesPullingTime="1.23005794s" totalInitContainerRuntime="66.293053ms" isStatefulPod=true podCreationTimestamp="2026-07-09 17:41:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:41.291158431 +0000 UTC m=+128.993659614" watchObservedRunningTime="2026-07-09 17:41:41.291995195 +0000 UTC m=+128.994496385" Jul 09 17:41:42.392826 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.392789 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:42.491300 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.491263 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:42.557694 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.557657 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:41:42.560109 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.560084 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86dbd8b7-a392-48da-901a-a23b1fabec38-metrics-certs\") pod \"network-metrics-daemon-7vlf2\" (UID: \"86dbd8b7-a392-48da-901a-a23b1fabec38\") " pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:41:42.632667 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.632638 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f99f5\"" Jul 09 17:41:42.641308 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.641280 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7vlf2" Jul 09 17:41:42.666399 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.666361 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:42.801885 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.801854 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7vlf2"] Jul 09 17:41:42.804815 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:41:42.804783 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dbd8b7_a392_48da_901a_a23b1fabec38.slice/crio-24ba5b0fe0b0f26fc6be87d15310cd9df5a920b82d0a2a73f4565ae96a1b6634 WatchSource:0}: Error finding container 24ba5b0fe0b0f26fc6be87d15310cd9df5a920b82d0a2a73f4565ae96a1b6634: Status 404 returned error can't find the container with id 24ba5b0fe0b0f26fc6be87d15310cd9df5a920b82d0a2a73f4565ae96a1b6634 Jul 09 17:41:42.809828 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.809801 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:42.879481 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:42.879410 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:43.281038 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:43.280997 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vlf2" event={"ID":"86dbd8b7-a392-48da-901a-a23b1fabec38","Type":"ContainerStarted","Data":"24ba5b0fe0b0f26fc6be87d15310cd9df5a920b82d0a2a73f4565ae96a1b6634"} Jul 09 17:41:43.947748 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:43.947708 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:44.286833 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:44.286797 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vlf2" event={"ID":"86dbd8b7-a392-48da-901a-a23b1fabec38","Type":"ContainerStarted","Data":"ff289491947a9cbebd235f9952f291faff930d363d2183d5b856a7401bbb3457"} Jul 09 17:41:44.286995 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:44.286858 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7vlf2" event={"ID":"86dbd8b7-a392-48da-901a-a23b1fabec38","Type":"ContainerStarted","Data":"10955eb28bb45f17604d9abb1f2aea1327df2b1cc2193d8c57ad59914ef4a7eb"} Jul 09 17:41:44.302772 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:44.302716 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7vlf2" podStartSLOduration=131.279084025 podStartE2EDuration="2m12.302696703s" totalImagesPullingTime="1.023612678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:44.301735767 +0000 UTC m=+132.004237004" watchObservedRunningTime="2026-07-09 17:41:44.302696703 +0000 UTC m=+132.005197967" Jul 09 17:41:44.441860 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:44.441811 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:44.512936 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:44.512902 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:45.267370 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:45.267337 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:45.324818 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:45.324786 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:48.040352 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:48.040317 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:48.101799 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:48.101768 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:51.151383 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:51.151343 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:51.249219 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:51.249182 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:51.424725 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:51.424635 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-94464db7-rhhq7"] Jul 09 17:41:51.424916 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:41:51.424897 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-94464db7-rhhq7" podUID="947050c5-395e-4330-8557-271ad164c107" Jul 09 17:41:51.506914 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:51.506877 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:51.605868 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:51.605831 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:52.313096 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.313065 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:41:52.320870 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.320838 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:41:52.442226 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442189 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-installation-pull-secrets\") pod \"947050c5-395e-4330-8557-271ad164c107\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " Jul 09 17:41:52.442226 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442224 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/947050c5-395e-4330-8557-271ad164c107-ca-trust-extracted\") pod \"947050c5-395e-4330-8557-271ad164c107\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " Jul 09 17:41:52.442521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442262 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5smj4\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-kube-api-access-5smj4\") pod \"947050c5-395e-4330-8557-271ad164c107\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " Jul 09 17:41:52.442521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442284 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-bound-sa-token\") pod \"947050c5-395e-4330-8557-271ad164c107\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " Jul 09 17:41:52.442521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442444 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-registry-certificates\") pod \"947050c5-395e-4330-8557-271ad164c107\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " Jul 09 17:41:52.442521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442506 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-image-registry-private-configuration\") pod \"947050c5-395e-4330-8557-271ad164c107\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " Jul 09 17:41:52.442715 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442568 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-trusted-ca\") pod \"947050c5-395e-4330-8557-271ad164c107\" (UID: \"947050c5-395e-4330-8557-271ad164c107\") " Jul 09 17:41:52.442715 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442580 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/947050c5-395e-4330-8557-271ad164c107-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "947050c5-395e-4330-8557-271ad164c107" (UID: "947050c5-395e-4330-8557-271ad164c107"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 17:41:52.442826 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442722 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "947050c5-395e-4330-8557-271ad164c107" (UID: "947050c5-395e-4330-8557-271ad164c107"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:41:52.442887 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442827 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-registry-certificates\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:41:52.442887 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.442847 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/947050c5-395e-4330-8557-271ad164c107-ca-trust-extracted\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:41:52.443021 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.443001 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "947050c5-395e-4330-8557-271ad164c107" (UID: "947050c5-395e-4330-8557-271ad164c107"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:41:52.444768 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.444734 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-kube-api-access-5smj4" (OuterVolumeSpecName: "kube-api-access-5smj4") pod "947050c5-395e-4330-8557-271ad164c107" (UID: "947050c5-395e-4330-8557-271ad164c107"). InnerVolumeSpecName "kube-api-access-5smj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:41:52.444768 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.444731 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "947050c5-395e-4330-8557-271ad164c107" (UID: "947050c5-395e-4330-8557-271ad164c107"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:41:52.444959 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.444787 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "947050c5-395e-4330-8557-271ad164c107" (UID: "947050c5-395e-4330-8557-271ad164c107"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:41:52.444959 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.444809 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "947050c5-395e-4330-8557-271ad164c107" (UID: "947050c5-395e-4330-8557-271ad164c107"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:41:52.543512 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.543454 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/947050c5-395e-4330-8557-271ad164c107-trusted-ca\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:41:52.543512 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.543505 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-installation-pull-secrets\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:41:52.543512 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.543517 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5smj4\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-kube-api-access-5smj4\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:41:52.543512 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.543526 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-bound-sa-token\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:41:52.543512 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:52.543535 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/947050c5-395e-4330-8557-271ad164c107-image-registry-private-configuration\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:41:53.315939 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.315896 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94464db7-rhhq7" Jul 09 17:41:53.355830 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.355796 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-94464db7-rhhq7"] Jul 09 17:41:53.360825 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.360798 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-94464db7-rhhq7"] Jul 09 17:41:53.451006 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.450969 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/947050c5-395e-4330-8557-271ad164c107-registry-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:41:53.866997 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.866953 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7448b84b6b-txjzf"] Jul 09 17:41:53.869942 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.869923 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:53.872181 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.872151 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 17:41:53.872363 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.872261 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 17:41:53.872363 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.872273 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 17:41:53.872656 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.872631 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 17:41:53.872798 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.872671 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 17:41:53.872798 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.872674 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 17:41:53.873155 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.873139 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pqnxv\"" Jul 09 17:41:53.873235 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.873171 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 17:41:53.878280 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.878260 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 09 17:41:53.883341 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.883319 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7448b84b6b-txjzf"] Jul 09 17:41:53.954193 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.954150 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-console-config\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:53.954193 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.954193 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-serving-cert\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:53.954447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.954216 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-service-ca\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:53.954447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.954293 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-oauth-config\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:53.954447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.954320 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-oauth-serving-cert\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:53.954447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.954343 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tndj8\" (UniqueName: \"kubernetes.io/projected/de24faf5-9659-4780-9c6c-29633599a64f-kube-api-access-tndj8\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:53.954447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:53.954360 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-trusted-ca-bundle\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.055697 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.055653 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-console-config\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.055921 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.055701 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-serving-cert\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.055921 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.055735 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-service-ca\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.055921 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.055776 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-oauth-config\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.055921 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.055797 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-oauth-serving-cert\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.055921 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.055851 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tndj8\" (UniqueName: \"kubernetes.io/projected/de24faf5-9659-4780-9c6c-29633599a64f-kube-api-access-tndj8\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.055921 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.055891 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-trusted-ca-bundle\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.056615 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.056584 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-console-config\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.056756 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.056599 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-service-ca\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.056756 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.056652 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-oauth-serving-cert\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.056877 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.056780 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-trusted-ca-bundle\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.058526 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.058496 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-serving-cert\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.058734 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.058712 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-oauth-config\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.064539 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.064515 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tndj8\" (UniqueName: \"kubernetes.io/projected/de24faf5-9659-4780-9c6c-29633599a64f-kube-api-access-tndj8\") pod \"console-7448b84b6b-txjzf\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.179848 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.179753 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:41:54.317664 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.317636 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7448b84b6b-txjzf"] Jul 09 17:41:54.321109 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:41:54.321080 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde24faf5_9659_4780_9c6c_29633599a64f.slice/crio-cf01cedf29b44264309dfe03445ce0b3c42bf89119502096dec0673d128270ef WatchSource:0}: Error finding container cf01cedf29b44264309dfe03445ce0b3c42bf89119502096dec0673d128270ef: Status 404 returned error can't find the container with id cf01cedf29b44264309dfe03445ce0b3c42bf89119502096dec0673d128270ef Jul 09 17:41:54.355009 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.354973 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:54.412692 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.412660 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:54.811171 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:54.811136 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="947050c5-395e-4330-8557-271ad164c107" path="/var/lib/kubelet/pods/947050c5-395e-4330-8557-271ad164c107/volumes" Jul 09 17:41:55.323936 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:55.323891 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7448b84b6b-txjzf" event={"ID":"de24faf5-9659-4780-9c6c-29633599a64f","Type":"ContainerStarted","Data":"cf01cedf29b44264309dfe03445ce0b3c42bf89119502096dec0673d128270ef"} Jul 09 17:41:56.251756 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:56.251675 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:56.315279 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:56.315241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:57.056034 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:57.056002 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:57.240705 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:57.240674 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:57.336226 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:57.336129 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7448b84b6b-txjzf" event={"ID":"de24faf5-9659-4780-9c6c-29633599a64f","Type":"ContainerStarted","Data":"e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3"} Jul 09 17:41:57.353458 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:57.353366 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7448b84b6b-txjzf" podStartSLOduration=1.471732375 podStartE2EDuration="4.353343279s" totalImagesPullingTime="2.881610904s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:41:57.352865154 +0000 UTC m=+145.055366351" watchObservedRunningTime="2026-07-09 17:41:57.353343279 +0000 UTC m=+145.055844471" Jul 09 17:41:58.209653 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:58.209617 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:58.271599 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:58.271562 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:41:59.910158 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:41:59.910123 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:00.018994 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:00.018960 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:04.180326 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:04.180278 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:42:04.180864 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:04.180464 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:42:04.185030 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:04.185004 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:42:04.364272 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:04.364237 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:42:04.525933 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:04.525871 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:04.588061 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:04.588022 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:04.646186 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:04.646155 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:10.761059 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:42:10.761007 2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6dtwp" podUID="40f09dde-c76e-410b-98cd-cdb1de2858d4" Jul 09 17:42:11.382213 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:11.382180 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dtwp" Jul 09 17:42:15.317896 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:15.317860 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:15.418549 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:15.418509 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:15.478259 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:15.478224 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:15.737085 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:15.737036 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:42:15.739464 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:15.739444 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/40f09dde-c76e-410b-98cd-cdb1de2858d4-metrics-tls\") pod \"dns-default-6dtwp\" (UID: \"40f09dde-c76e-410b-98cd-cdb1de2858d4\") " pod="openshift-dns/dns-default-6dtwp" Jul 09 17:42:15.885152 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:15.885115 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lljhl\"" Jul 09 17:42:15.893413 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:15.893379 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6dtwp" Jul 09 17:42:16.048056 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:16.047951 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6dtwp"] Jul 09 17:42:16.051536 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:42:16.051497 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f09dde_c76e_410b_98cd_cdb1de2858d4.slice/crio-ac081b4c98208c75776d0560f9e63cfeeac8144c7c6c5dd23ae0a58115feae8c WatchSource:0}: Error finding container ac081b4c98208c75776d0560f9e63cfeeac8144c7c6c5dd23ae0a58115feae8c: Status 404 returned error can't find the container with id ac081b4c98208c75776d0560f9e63cfeeac8144c7c6c5dd23ae0a58115feae8c Jul 09 17:42:16.398928 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:16.398835 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dtwp" event={"ID":"40f09dde-c76e-410b-98cd-cdb1de2858d4","Type":"ContainerStarted","Data":"ac081b4c98208c75776d0560f9e63cfeeac8144c7c6c5dd23ae0a58115feae8c"} Jul 09 17:42:18.408919 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:18.408805 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dtwp" event={"ID":"40f09dde-c76e-410b-98cd-cdb1de2858d4","Type":"ContainerStarted","Data":"1d62f7831f2a11dce817a635612d83dadd0bfc384d17bff7cb214345f2fcff4d"} Jul 09 17:42:18.408919 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:18.408876 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6dtwp" event={"ID":"40f09dde-c76e-410b-98cd-cdb1de2858d4","Type":"ContainerStarted","Data":"a7bcca474fd9ba0d809b3b9f415ad057839375fa2c42345e3a010936c1477c33"} Jul 09 17:42:18.409458 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:18.408935 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6dtwp" Jul 09 17:42:18.426185 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:18.426119 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6dtwp" podStartSLOduration=129.403096857 podStartE2EDuration="2m11.426095623s" totalImagesPullingTime="2.022998766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:42:18.425534088 +0000 UTC m=+166.128035278" watchObservedRunningTime="2026-07-09 17:42:18.426095623 +0000 UTC m=+166.128596812" Jul 09 17:42:22.227129 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:22.227087 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:22.279858 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:22.279825 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:28.414407 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:28.414371 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6dtwp" Jul 09 17:42:37.476753 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:37.476715 2560 generic.go:356] "Generic (PLEG): container finished" podID="62d18c03-835d-48a0-86e6-7a1f9f0d9882" containerID="e4a7d9acb79bb5a8a3d13d1a843bd5e9c2071270001fca498244a2078f10b3f8" exitCode=0 Jul 09 17:42:37.477140 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:37.476780 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" event={"ID":"62d18c03-835d-48a0-86e6-7a1f9f0d9882","Type":"ContainerDied","Data":"e4a7d9acb79bb5a8a3d13d1a843bd5e9c2071270001fca498244a2078f10b3f8"} Jul 09 17:42:37.477140 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:37.477082 2560 scope.go:117] "RemoveContainer" containerID="e4a7d9acb79bb5a8a3d13d1a843bd5e9c2071270001fca498244a2078f10b3f8" Jul 09 17:42:37.564500 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:37.564467 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:37.617934 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:37.617900 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:38.481769 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:38.481729 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-cvvnb" event={"ID":"62d18c03-835d-48a0-86e6-7a1f9f0d9882","Type":"ContainerStarted","Data":"cc9a23517deec2e3750404a3675137535f76f06bcf0d3cbd875907cc0634fcfa"} Jul 09 17:42:53.774950 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:53.774915 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:53.838704 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:53.838662 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:56.590467 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.590410 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f4fd6db6f-2gg5j"] Jul 09 17:42:56.593725 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.593698 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.606492 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.606460 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4fd6db6f-2gg5j"] Jul 09 17:42:56.648525 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.648485 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:56.760782 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.760750 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-trusted-ca-bundle\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.760782 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.760784 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-oauth-serving-cert\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.761040 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.760809 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-oauth-config\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.761040 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.760834 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-config\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.761040 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.760898 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcs2c\" (UniqueName: \"kubernetes.io/projected/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-kube-api-access-vcs2c\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.761040 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.760921 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-service-ca\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.761040 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.760969 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-serving-cert\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.789366 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.789330 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:56.862102 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862016 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcs2c\" (UniqueName: \"kubernetes.io/projected/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-kube-api-access-vcs2c\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.862102 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862060 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-service-ca\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.862102 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862090 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-serving-cert\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.862380 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862124 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-trusted-ca-bundle\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.862380 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862147 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-oauth-serving-cert\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.862380 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862183 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-oauth-config\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.862380 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862212 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-config\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.862940 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862907 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-oauth-serving-cert\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.863126 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.862983 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-config\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.863126 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.863115 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-service-ca\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.863306 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.863285 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-trusted-ca-bundle\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.864770 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.864748 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-oauth-config\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.864899 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.864884 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-serving-cert\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.871060 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.871031 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcs2c\" (UniqueName: \"kubernetes.io/projected/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-kube-api-access-vcs2c\") pod \"console-f4fd6db6f-2gg5j\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:56.891004 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.890973 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:56.902935 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:56.902902 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:42:57.042445 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:57.042395 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f4fd6db6f-2gg5j"] Jul 09 17:42:57.047084 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:42:57.047050 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84b77b2_3bd3_4e77_91d7_bf6b2fe11c75.slice/crio-df0aceace60170212fb32cb0454788c3a4bd71863ff82a6e256af4ff8b5325ce WatchSource:0}: Error finding container df0aceace60170212fb32cb0454788c3a4bd71863ff82a6e256af4ff8b5325ce: Status 404 returned error can't find the container with id df0aceace60170212fb32cb0454788c3a4bd71863ff82a6e256af4ff8b5325ce Jul 09 17:42:57.387989 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:57.387945 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:57.439703 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:57.439668 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:42:57.548567 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:57.548522 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4fd6db6f-2gg5j" event={"ID":"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75","Type":"ContainerStarted","Data":"358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98"} Jul 09 17:42:57.548741 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:57.548589 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4fd6db6f-2gg5j" event={"ID":"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75","Type":"ContainerStarted","Data":"df0aceace60170212fb32cb0454788c3a4bd71863ff82a6e256af4ff8b5325ce"} Jul 09 17:42:57.566221 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:42:57.566171 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f4fd6db6f-2gg5j" podStartSLOduration=1.5661536470000001 podStartE2EDuration="1.566153647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:42:57.565168704 +0000 UTC m=+205.267669900" watchObservedRunningTime="2026-07-09 17:42:57.566153647 +0000 UTC m=+205.268654836" Jul 09 17:43:05.691538 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:05.691497 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:05.749444 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:05.749389 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:06.903749 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:06.903697 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:43:06.903749 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:06.903768 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:43:06.908581 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:06.908555 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:43:07.589303 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:07.589268 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:43:07.638265 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:07.638233 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7448b84b6b-txjzf"] Jul 09 17:43:07.753884 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:07.753855 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:07.852314 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:07.852227 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:07.914747 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:07.914712 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:20.398705 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:20.398669 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:22.249428 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:22.249377 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:22.349470 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:22.349438 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:32.664633 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.664565 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7448b84b6b-txjzf" podUID="de24faf5-9659-4780-9c6c-29633599a64f" containerName="console" containerID="cri-o://e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3" gracePeriod=15 Jul 09 17:43:32.910490 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.910464 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:43:32.937411 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.937315 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-service-ca\") pod \"de24faf5-9659-4780-9c6c-29633599a64f\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " Jul 09 17:43:32.937717 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.937693 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-trusted-ca-bundle\") pod \"de24faf5-9659-4780-9c6c-29633599a64f\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " Jul 09 17:43:32.937821 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.937790 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tndj8\" (UniqueName: \"kubernetes.io/projected/de24faf5-9659-4780-9c6c-29633599a64f-kube-api-access-tndj8\") pod \"de24faf5-9659-4780-9c6c-29633599a64f\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " Jul 09 17:43:32.937881 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.937822 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-console-config\") pod \"de24faf5-9659-4780-9c6c-29633599a64f\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " Jul 09 17:43:32.937881 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.937848 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-serving-cert\") pod \"de24faf5-9659-4780-9c6c-29633599a64f\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " Jul 09 17:43:32.937981 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.937881 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-oauth-serving-cert\") pod \"de24faf5-9659-4780-9c6c-29633599a64f\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " Jul 09 17:43:32.937981 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.937891 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-service-ca" (OuterVolumeSpecName: "service-ca") pod "de24faf5-9659-4780-9c6c-29633599a64f" (UID: "de24faf5-9659-4780-9c6c-29633599a64f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:43:32.937981 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.937930 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-oauth-config\") pod \"de24faf5-9659-4780-9c6c-29633599a64f\" (UID: \"de24faf5-9659-4780-9c6c-29633599a64f\") " Jul 09 17:43:32.938157 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.938137 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-service-ca\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:43:32.938272 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.938204 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-console-config" (OuterVolumeSpecName: "console-config") pod "de24faf5-9659-4780-9c6c-29633599a64f" (UID: "de24faf5-9659-4780-9c6c-29633599a64f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:43:32.938562 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.938532 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "de24faf5-9659-4780-9c6c-29633599a64f" (UID: "de24faf5-9659-4780-9c6c-29633599a64f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:43:32.938672 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.938579 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "de24faf5-9659-4780-9c6c-29633599a64f" (UID: "de24faf5-9659-4780-9c6c-29633599a64f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:43:32.940772 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.940745 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "de24faf5-9659-4780-9c6c-29633599a64f" (UID: "de24faf5-9659-4780-9c6c-29633599a64f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:43:32.940870 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.940785 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de24faf5-9659-4780-9c6c-29633599a64f-kube-api-access-tndj8" (OuterVolumeSpecName: "kube-api-access-tndj8") pod "de24faf5-9659-4780-9c6c-29633599a64f" (UID: "de24faf5-9659-4780-9c6c-29633599a64f"). InnerVolumeSpecName "kube-api-access-tndj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:43:32.940870 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:32.940804 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "de24faf5-9659-4780-9c6c-29633599a64f" (UID: "de24faf5-9659-4780-9c6c-29633599a64f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:43:33.039216 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.039180 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-trusted-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:43:33.039216 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.039211 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tndj8\" (UniqueName: \"kubernetes.io/projected/de24faf5-9659-4780-9c6c-29633599a64f-kube-api-access-tndj8\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:43:33.039216 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.039223 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-console-config\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:43:33.039511 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.039231 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-serving-cert\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:43:33.039511 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.039244 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/de24faf5-9659-4780-9c6c-29633599a64f-oauth-serving-cert\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:43:33.039511 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.039255 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/de24faf5-9659-4780-9c6c-29633599a64f-console-oauth-config\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:43:33.676394 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.676360 2560 generic.go:356] "Generic (PLEG): container finished" podID="de24faf5-9659-4780-9c6c-29633599a64f" containerID="e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3" exitCode=0 Jul 09 17:43:33.676841 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.676410 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7448b84b6b-txjzf" Jul 09 17:43:33.676841 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.676459 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7448b84b6b-txjzf" event={"ID":"de24faf5-9659-4780-9c6c-29633599a64f","Type":"ContainerDied","Data":"e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3"} Jul 09 17:43:33.676841 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.676517 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7448b84b6b-txjzf" event={"ID":"de24faf5-9659-4780-9c6c-29633599a64f","Type":"ContainerDied","Data":"cf01cedf29b44264309dfe03445ce0b3c42bf89119502096dec0673d128270ef"} Jul 09 17:43:33.676841 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.676539 2560 scope.go:117] "RemoveContainer" containerID="e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3" Jul 09 17:43:33.686520 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.686498 2560 scope.go:117] "RemoveContainer" containerID="e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3" Jul 09 17:43:33.686810 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:43:33.686792 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3\": container with ID starting with e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3 not found: ID does not exist" containerID="e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3" Jul 09 17:43:33.686860 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.686818 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3"} err="failed to get container status \"e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3\": rpc error: code = NotFound desc = could not find container \"e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3\": container with ID starting with e521ac4a001c318178fb6473f07de54a093ad3f3bff5429430ab8e124a0b8ce3 not found: ID does not exist" Jul 09 17:43:33.697104 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.697070 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7448b84b6b-txjzf"] Jul 09 17:43:33.700851 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:33.700822 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7448b84b6b-txjzf"] Jul 09 17:43:34.811442 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:34.811374 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de24faf5-9659-4780-9c6c-29633599a64f" path="/var/lib/kubelet/pods/de24faf5-9659-4780-9c6c-29633599a64f/volumes" Jul 09 17:43:49.685009 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:49.684967 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:43:49.734436 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:43:49.734390 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:08.371180 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:08.371094 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:08.466207 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:08.466177 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:08.730371 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:08.730325 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:08.836445 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:08.836393 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:08.933283 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:08.933251 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:32.694603 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:32.694568 2560 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 17:44:32.807325 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:32.807149 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:36.123525 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:36.123480 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:36.219474 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:36.219408 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:37.748947 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.748911 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv"] Jul 09 17:44:37.749336 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.749171 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de24faf5-9659-4780-9c6c-29633599a64f" containerName="console" Jul 09 17:44:37.749336 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.749181 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="de24faf5-9659-4780-9c6c-29633599a64f" containerName="console" Jul 09 17:44:37.749336 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.749231 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="de24faf5-9659-4780-9c6c-29633599a64f" containerName="console" Jul 09 17:44:37.752055 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.752036 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:37.755220 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.755198 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Jul 09 17:44:37.755376 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.755358 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Jul 09 17:44:37.755558 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.755539 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-7q856\"" Jul 09 17:44:37.755635 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.755618 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Jul 09 17:44:37.766857 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.766825 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv"] Jul 09 17:44:37.825811 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.825771 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grwdb\" (UniqueName: \"kubernetes.io/projected/08533d97-2bf9-431d-8b76-aeeebdedbfba-kube-api-access-grwdb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv\" (UID: \"08533d97-2bf9-431d-8b76-aeeebdedbfba\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:37.825811 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.825816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/08533d97-2bf9-431d-8b76-aeeebdedbfba-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv\" (UID: \"08533d97-2bf9-431d-8b76-aeeebdedbfba\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:37.927006 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.926971 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/08533d97-2bf9-431d-8b76-aeeebdedbfba-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv\" (UID: \"08533d97-2bf9-431d-8b76-aeeebdedbfba\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:37.927170 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.927044 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grwdb\" (UniqueName: \"kubernetes.io/projected/08533d97-2bf9-431d-8b76-aeeebdedbfba-kube-api-access-grwdb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv\" (UID: \"08533d97-2bf9-431d-8b76-aeeebdedbfba\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:37.929391 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.929358 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/08533d97-2bf9-431d-8b76-aeeebdedbfba-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv\" (UID: \"08533d97-2bf9-431d-8b76-aeeebdedbfba\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:37.936491 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:37.936461 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grwdb\" (UniqueName: \"kubernetes.io/projected/08533d97-2bf9-431d-8b76-aeeebdedbfba-kube-api-access-grwdb\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv\" (UID: \"08533d97-2bf9-431d-8b76-aeeebdedbfba\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:38.062468 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:38.062341 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:38.204437 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:38.204380 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv"] Jul 09 17:44:38.207861 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:44:38.207833 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08533d97_2bf9_431d_8b76_aeeebdedbfba.slice/crio-90a810cab3ee08e7bd887fd133ebd72cb2ffd4805ade9b8515d9baf3f779caa3 WatchSource:0}: Error finding container 90a810cab3ee08e7bd887fd133ebd72cb2ffd4805ade9b8515d9baf3f779caa3: Status 404 returned error can't find the container with id 90a810cab3ee08e7bd887fd133ebd72cb2ffd4805ade9b8515d9baf3f779caa3 Jul 09 17:44:38.209665 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:38.209648 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 17:44:38.901877 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:38.901844 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" event={"ID":"08533d97-2bf9-431d-8b76-aeeebdedbfba","Type":"ContainerStarted","Data":"90a810cab3ee08e7bd887fd133ebd72cb2ffd4805ade9b8515d9baf3f779caa3"} Jul 09 17:44:42.447623 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:42.447546 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:42.738355 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:42.738241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:42.866364 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:42.866327 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:42.923311 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:42.923263 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" event={"ID":"08533d97-2bf9-431d-8b76-aeeebdedbfba","Type":"ContainerStarted","Data":"71e555d2d33a772b70ff1b8f892ffd31da086839d7a2bd9487caad86ba9fbeb3"} Jul 09 17:44:42.923533 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:42.923391 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:44:42.942163 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:42.942126 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:42.942690 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:42.942646 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" podStartSLOduration=2.230638677 podStartE2EDuration="5.942631702s" totalImagesPullingTime="3.711993025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:44:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:44:42.941309701 +0000 UTC m=+310.643810895" watchObservedRunningTime="2026-07-09 17:44:42.942631702 +0000 UTC m=+310.645132892" Jul 09 17:44:43.185114 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.185007 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:43.284562 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.284525 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:43.312846 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.312811 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jnc29"] Jul 09 17:44:43.315922 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.315902 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:43.318394 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.318363 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-mgk2l\"" Jul 09 17:44:43.318533 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.318363 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Jul 09 17:44:43.318634 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.318614 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\"" Jul 09 17:44:43.324402 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.324376 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jnc29"] Jul 09 17:44:43.370538 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.370490 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/59976e07-8a2d-4031-b1de-fbe682a57027-certificates\") pod \"keda-admission-5fbd5c4644-jnc29\" (UID: \"59976e07-8a2d-4031-b1de-fbe682a57027\") " pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:43.370775 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.370549 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6f4p\" (UniqueName: \"kubernetes.io/projected/59976e07-8a2d-4031-b1de-fbe682a57027-kube-api-access-l6f4p\") pod \"keda-admission-5fbd5c4644-jnc29\" (UID: \"59976e07-8a2d-4031-b1de-fbe682a57027\") " pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:43.395044 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.395010 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:43.471501 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.471464 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/59976e07-8a2d-4031-b1de-fbe682a57027-certificates\") pod \"keda-admission-5fbd5c4644-jnc29\" (UID: \"59976e07-8a2d-4031-b1de-fbe682a57027\") " pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:43.471885 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.471524 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6f4p\" (UniqueName: \"kubernetes.io/projected/59976e07-8a2d-4031-b1de-fbe682a57027-kube-api-access-l6f4p\") pod \"keda-admission-5fbd5c4644-jnc29\" (UID: \"59976e07-8a2d-4031-b1de-fbe682a57027\") " pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:43.474065 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.474038 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/59976e07-8a2d-4031-b1de-fbe682a57027-certificates\") pod \"keda-admission-5fbd5c4644-jnc29\" (UID: \"59976e07-8a2d-4031-b1de-fbe682a57027\") " pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:43.479748 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.479716 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6f4p\" (UniqueName: \"kubernetes.io/projected/59976e07-8a2d-4031-b1de-fbe682a57027-kube-api-access-l6f4p\") pod \"keda-admission-5fbd5c4644-jnc29\" (UID: \"59976e07-8a2d-4031-b1de-fbe682a57027\") " pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:43.627585 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.627549 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:43.765780 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.765702 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-jnc29"] Jul 09 17:44:43.769018 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:44:43.768987 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59976e07_8a2d_4031_b1de_fbe682a57027.slice/crio-da11ce8f10278a3bdd87f1a2461ec221193d7913e669c025f5e34e1705ca915e WatchSource:0}: Error finding container da11ce8f10278a3bdd87f1a2461ec221193d7913e669c025f5e34e1705ca915e: Status 404 returned error can't find the container with id da11ce8f10278a3bdd87f1a2461ec221193d7913e669c025f5e34e1705ca915e Jul 09 17:44:43.815516 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.815480 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:43.928375 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.928341 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" event={"ID":"59976e07-8a2d-4031-b1de-fbe682a57027","Type":"ContainerStarted","Data":"da11ce8f10278a3bdd87f1a2461ec221193d7913e669c025f5e34e1705ca915e"} Jul 09 17:44:43.986355 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:43.986314 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:44.164891 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:44.164804 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:44.255151 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:44.255109 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:44:45.940114 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:45.940081 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" event={"ID":"59976e07-8a2d-4031-b1de-fbe682a57027","Type":"ContainerStarted","Data":"f1ad0a8b270ae9051ef63733e75c1cc3f56e25ae9c9bd7247be5d2542b05f75e"} Jul 09 17:44:45.940597 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:45.940177 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:44:45.962876 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:44:45.962813 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" podStartSLOduration=1.35704703 podStartE2EDuration="2.962790147s" totalImagesPullingTime="1.605743117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:44:45.961238384 +0000 UTC m=+313.663739573" watchObservedRunningTime="2026-07-09 17:44:45.962790147 +0000 UTC m=+313.665291339" Jul 09 17:45:03.930910 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:03.930871 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-xhfkv" Jul 09 17:45:06.946200 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:06.946170 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-jnc29" Jul 09 17:45:09.451874 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:09.451838 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:45:09.538967 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:09.538931 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:45:58.023952 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.023903 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:45:58.147092 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.147049 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-x5h68"] Jul 09 17:45:58.149400 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.149378 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-qxhqm"] Jul 09 17:45:58.149525 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.149505 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:58.151791 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.151769 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:45:58.155114 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.155054 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Jul 09 17:45:58.155244 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.155121 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-gflfn\"" Jul 09 17:45:58.155342 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.155303 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Jul 09 17:45:58.155511 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.155484 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Jul 09 17:45:58.155631 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.155573 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-5mx7v\"" Jul 09 17:45:58.156395 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.156380 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Jul 09 17:45:58.174546 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.174518 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-x5h68"] Jul 09 17:45:58.176226 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.176201 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-qxhqm"] Jul 09 17:45:58.245270 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.245228 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-cert\") pod \"llmisvc-controller-manager-654c557856-x5h68\" (UID: \"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2\") " pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:58.245490 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.245282 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dccwq\" (UniqueName: \"kubernetes.io/projected/5b1b42de-7ded-4077-aa99-ea495fa437fa-kube-api-access-dccwq\") pod \"kserve-controller-manager-6b8548bc6d-qxhqm\" (UID: \"5b1b42de-7ded-4077-aa99-ea495fa437fa\") " pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:45:58.245490 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.245301 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t59cn\" (UniqueName: \"kubernetes.io/projected/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-kube-api-access-t59cn\") pod \"llmisvc-controller-manager-654c557856-x5h68\" (UID: \"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2\") " pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:58.245490 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.245388 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1b42de-7ded-4077-aa99-ea495fa437fa-cert\") pod \"kserve-controller-manager-6b8548bc6d-qxhqm\" (UID: \"5b1b42de-7ded-4077-aa99-ea495fa437fa\") " pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:45:58.346588 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.346487 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1b42de-7ded-4077-aa99-ea495fa437fa-cert\") pod \"kserve-controller-manager-6b8548bc6d-qxhqm\" (UID: \"5b1b42de-7ded-4077-aa99-ea495fa437fa\") " pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:45:58.346588 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.346550 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-cert\") pod \"llmisvc-controller-manager-654c557856-x5h68\" (UID: \"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2\") " pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:58.346841 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:45:58.346641 2560 secret.go:189] Couldn't get secret kserve/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 09 17:45:58.346841 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:45:58.346704 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-cert podName:b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2 nodeName:}" failed. No retries permitted until 2026-07-09 17:45:58.846684641 +0000 UTC m=+386.549185809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-cert") pod "llmisvc-controller-manager-654c557856-x5h68" (UID: "b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2") : secret "llmisvc-webhook-server-cert" not found Jul 09 17:45:58.346841 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.346732 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dccwq\" (UniqueName: \"kubernetes.io/projected/5b1b42de-7ded-4077-aa99-ea495fa437fa-kube-api-access-dccwq\") pod \"kserve-controller-manager-6b8548bc6d-qxhqm\" (UID: \"5b1b42de-7ded-4077-aa99-ea495fa437fa\") " pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:45:58.346841 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.346776 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t59cn\" (UniqueName: \"kubernetes.io/projected/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-kube-api-access-t59cn\") pod \"llmisvc-controller-manager-654c557856-x5h68\" (UID: \"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2\") " pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:58.349236 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.349217 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1b42de-7ded-4077-aa99-ea495fa437fa-cert\") pod \"kserve-controller-manager-6b8548bc6d-qxhqm\" (UID: \"5b1b42de-7ded-4077-aa99-ea495fa437fa\") " pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:45:58.368028 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.367995 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t59cn\" (UniqueName: \"kubernetes.io/projected/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-kube-api-access-t59cn\") pod \"llmisvc-controller-manager-654c557856-x5h68\" (UID: \"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2\") " pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:58.370777 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.370750 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dccwq\" (UniqueName: \"kubernetes.io/projected/5b1b42de-7ded-4077-aa99-ea495fa437fa-kube-api-access-dccwq\") pod \"kserve-controller-manager-6b8548bc6d-qxhqm\" (UID: \"5b1b42de-7ded-4077-aa99-ea495fa437fa\") " pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:45:58.471546 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.471508 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:45:58.629583 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.629371 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-qxhqm"] Jul 09 17:45:58.632377 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:45:58.632347 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1b42de_7ded_4077_aa99_ea495fa437fa.slice/crio-e5082c0b4d404ef1807d16ef62380d23e6831d09359f8217d2e7a16342383e0f WatchSource:0}: Error finding container e5082c0b4d404ef1807d16ef62380d23e6831d09359f8217d2e7a16342383e0f: Status 404 returned error can't find the container with id e5082c0b4d404ef1807d16ef62380d23e6831d09359f8217d2e7a16342383e0f Jul 09 17:45:58.850785 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.850742 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-cert\") pod \"llmisvc-controller-manager-654c557856-x5h68\" (UID: \"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2\") " pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:58.853292 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:58.853269 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2-cert\") pod \"llmisvc-controller-manager-654c557856-x5h68\" (UID: \"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2\") " pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:59.062650 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:59.062608 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:45:59.201121 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:59.201089 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-654c557856-x5h68"] Jul 09 17:45:59.203736 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:59.203700 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" event={"ID":"5b1b42de-7ded-4077-aa99-ea495fa437fa","Type":"ContainerStarted","Data":"e5082c0b4d404ef1807d16ef62380d23e6831d09359f8217d2e7a16342383e0f"} Jul 09 17:45:59.204526 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:45:59.204500 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb4fb0bcc_5a67_4a91_9183_f53c40cd1cd2.slice/crio-85a728ae776b2a8cebc0d332c8d80c1b27705dc15c5712ac4c298c736fb4daa9 WatchSource:0}: Error finding container 85a728ae776b2a8cebc0d332c8d80c1b27705dc15c5712ac4c298c736fb4daa9: Status 404 returned error can't find the container with id 85a728ae776b2a8cebc0d332c8d80c1b27705dc15c5712ac4c298c736fb4daa9 Jul 09 17:45:59.351968 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:45:59.351866 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:46:00.210352 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:00.210300 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-x5h68" event={"ID":"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2","Type":"ContainerStarted","Data":"85a728ae776b2a8cebc0d332c8d80c1b27705dc15c5712ac4c298c736fb4daa9"} Jul 09 17:46:02.219886 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:02.219844 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" event={"ID":"5b1b42de-7ded-4077-aa99-ea495fa437fa","Type":"ContainerStarted","Data":"2612942fc08502a5a1cb7770a1c9ef98514924a20bd2588b661f85feac4cb3bb"} Jul 09 17:46:02.220365 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:02.220110 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:46:02.238397 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:02.238336 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" podStartSLOduration=1.212989805 podStartE2EDuration="4.2383167s" totalImagesPullingTime="3.025326895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:46:02.237792961 +0000 UTC m=+389.940294149" watchObservedRunningTime="2026-07-09 17:46:02.2383167 +0000 UTC m=+389.940817889" Jul 09 17:46:04.229877 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:04.229843 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-654c557856-x5h68" event={"ID":"b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2","Type":"ContainerStarted","Data":"e262e658fbb01cf065e3e76802dcf33f2f97fbbe383bb7b9edd6c16072c7bde1"} Jul 09 17:46:04.230297 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:04.229928 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:46:04.253835 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:04.253784 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-654c557856-x5h68" podStartSLOduration=1.867735627 podStartE2EDuration="6.253765619s" totalImagesPullingTime="4.386029992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:46:04.251470323 +0000 UTC m=+391.953971513" watchObservedRunningTime="2026-07-09 17:46:04.253765619 +0000 UTC m=+391.956266809" Jul 09 17:46:29.153462 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:29.153429 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:46:29.224748 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:29.224710 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:46:33.229479 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:33.229443 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:46:35.239089 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:35.239052 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-654c557856-x5h68" Jul 09 17:46:37.140782 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.140742 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-qxhqm"] Jul 09 17:46:37.141269 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.140959 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" podUID="5b1b42de-7ded-4077-aa99-ea495fa437fa" containerName="manager" containerID="cri-o://2612942fc08502a5a1cb7770a1c9ef98514924a20bd2588b661f85feac4cb3bb" gracePeriod=10 Jul 09 17:46:37.220247 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.220212 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-9ff4t"] Jul 09 17:46:37.222318 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.222302 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:37.243549 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.243514 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-9ff4t"] Jul 09 17:46:37.358744 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.358704 2560 generic.go:356] "Generic (PLEG): container finished" podID="5b1b42de-7ded-4077-aa99-ea495fa437fa" containerID="2612942fc08502a5a1cb7770a1c9ef98514924a20bd2588b661f85feac4cb3bb" exitCode=0 Jul 09 17:46:37.358945 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.358775 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" event={"ID":"5b1b42de-7ded-4077-aa99-ea495fa437fa","Type":"ContainerDied","Data":"2612942fc08502a5a1cb7770a1c9ef98514924a20bd2588b661f85feac4cb3bb"} Jul 09 17:46:37.378314 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.378287 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20a09a93-93d8-4a3c-826b-9dbe3b0f1792-cert\") pod \"kserve-controller-manager-6b8548bc6d-9ff4t\" (UID: \"20a09a93-93d8-4a3c-826b-9dbe3b0f1792\") " pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:37.378475 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.378356 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl9r7\" (UniqueName: \"kubernetes.io/projected/20a09a93-93d8-4a3c-826b-9dbe3b0f1792-kube-api-access-pl9r7\") pod \"kserve-controller-manager-6b8548bc6d-9ff4t\" (UID: \"20a09a93-93d8-4a3c-826b-9dbe3b0f1792\") " pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:37.382780 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.382760 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:46:37.478852 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.478818 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dccwq\" (UniqueName: \"kubernetes.io/projected/5b1b42de-7ded-4077-aa99-ea495fa437fa-kube-api-access-dccwq\") pod \"5b1b42de-7ded-4077-aa99-ea495fa437fa\" (UID: \"5b1b42de-7ded-4077-aa99-ea495fa437fa\") " Jul 09 17:46:37.479034 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.478885 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1b42de-7ded-4077-aa99-ea495fa437fa-cert\") pod \"5b1b42de-7ded-4077-aa99-ea495fa437fa\" (UID: \"5b1b42de-7ded-4077-aa99-ea495fa437fa\") " Jul 09 17:46:37.479034 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.478968 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20a09a93-93d8-4a3c-826b-9dbe3b0f1792-cert\") pod \"kserve-controller-manager-6b8548bc6d-9ff4t\" (UID: \"20a09a93-93d8-4a3c-826b-9dbe3b0f1792\") " pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:37.479034 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.479027 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl9r7\" (UniqueName: \"kubernetes.io/projected/20a09a93-93d8-4a3c-826b-9dbe3b0f1792-kube-api-access-pl9r7\") pod \"kserve-controller-manager-6b8548bc6d-9ff4t\" (UID: \"20a09a93-93d8-4a3c-826b-9dbe3b0f1792\") " pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:37.481182 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.481149 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1b42de-7ded-4077-aa99-ea495fa437fa-cert" (OuterVolumeSpecName: "cert") pod "5b1b42de-7ded-4077-aa99-ea495fa437fa" (UID: "5b1b42de-7ded-4077-aa99-ea495fa437fa"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:46:37.481182 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.481153 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b1b42de-7ded-4077-aa99-ea495fa437fa-kube-api-access-dccwq" (OuterVolumeSpecName: "kube-api-access-dccwq") pod "5b1b42de-7ded-4077-aa99-ea495fa437fa" (UID: "5b1b42de-7ded-4077-aa99-ea495fa437fa"). InnerVolumeSpecName "kube-api-access-dccwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:46:37.481342 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.481265 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/20a09a93-93d8-4a3c-826b-9dbe3b0f1792-cert\") pod \"kserve-controller-manager-6b8548bc6d-9ff4t\" (UID: \"20a09a93-93d8-4a3c-826b-9dbe3b0f1792\") " pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:37.493677 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.493645 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl9r7\" (UniqueName: \"kubernetes.io/projected/20a09a93-93d8-4a3c-826b-9dbe3b0f1792-kube-api-access-pl9r7\") pod \"kserve-controller-manager-6b8548bc6d-9ff4t\" (UID: \"20a09a93-93d8-4a3c-826b-9dbe3b0f1792\") " pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:37.532537 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.532493 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:37.580290 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.580255 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dccwq\" (UniqueName: \"kubernetes.io/projected/5b1b42de-7ded-4077-aa99-ea495fa437fa-kube-api-access-dccwq\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:46:37.580290 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.580287 2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1b42de-7ded-4077-aa99-ea495fa437fa-cert\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:46:37.675532 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:37.675497 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-9ff4t"] Jul 09 17:46:37.678872 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:46:37.678837 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a09a93_93d8_4a3c_826b_9dbe3b0f1792.slice/crio-dd6d4c46e6e3be7018fcc5a3dc680d5eba3dde8531420ef63c1d7951d00621fd WatchSource:0}: Error finding container dd6d4c46e6e3be7018fcc5a3dc680d5eba3dde8531420ef63c1d7951d00621fd: Status 404 returned error can't find the container with id dd6d4c46e6e3be7018fcc5a3dc680d5eba3dde8531420ef63c1d7951d00621fd Jul 09 17:46:38.363755 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:38.363722 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" Jul 09 17:46:38.363755 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:38.363728 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b8548bc6d-qxhqm" event={"ID":"5b1b42de-7ded-4077-aa99-ea495fa437fa","Type":"ContainerDied","Data":"e5082c0b4d404ef1807d16ef62380d23e6831d09359f8217d2e7a16342383e0f"} Jul 09 17:46:38.364263 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:38.363788 2560 scope.go:117] "RemoveContainer" containerID="2612942fc08502a5a1cb7770a1c9ef98514924a20bd2588b661f85feac4cb3bb" Jul 09 17:46:38.365052 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:38.365007 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" event={"ID":"20a09a93-93d8-4a3c-826b-9dbe3b0f1792","Type":"ContainerStarted","Data":"dd6d4c46e6e3be7018fcc5a3dc680d5eba3dde8531420ef63c1d7951d00621fd"} Jul 09 17:46:38.391319 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:38.391281 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-qxhqm"] Jul 09 17:46:38.401115 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:38.401082 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6b8548bc6d-qxhqm"] Jul 09 17:46:38.811896 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:38.811863 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1b42de-7ded-4077-aa99-ea495fa437fa" path="/var/lib/kubelet/pods/5b1b42de-7ded-4077-aa99-ea495fa437fa/volumes" Jul 09 17:46:40.375116 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:40.375082 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" event={"ID":"20a09a93-93d8-4a3c-826b-9dbe3b0f1792","Type":"ContainerStarted","Data":"5126fc7db2a8ab9b10253b9fa241ace3f93989e9c5e6f68721ae6d589bb1c4d8"} Jul 09 17:46:40.375562 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:40.375187 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:46:40.397813 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:46:40.397758 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" podStartSLOduration=1.648181669 podStartE2EDuration="3.397737593s" totalImagesPullingTime="1.749555924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:46:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:46:40.395556632 +0000 UTC m=+428.098057822" watchObservedRunningTime="2026-07-09 17:46:40.397737593 +0000 UTC m=+428.100238825" Jul 09 17:47:02.228924 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.228835 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-57485bcffb-lk47g"] Jul 09 17:47:02.229317 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.229157 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1b42de-7ded-4077-aa99-ea495fa437fa" containerName="manager" Jul 09 17:47:02.229317 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.229169 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1b42de-7ded-4077-aa99-ea495fa437fa" containerName="manager" Jul 09 17:47:02.229317 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.229224 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1b42de-7ded-4077-aa99-ea495fa437fa" containerName="manager" Jul 09 17:47:02.232081 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.232062 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.246610 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.246583 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57485bcffb-lk47g"] Jul 09 17:47:02.254594 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.254565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-service-ca\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.254594 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.254607 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7576a070-9722-44b8-8331-63f928ed6b68-console-serving-cert\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.254827 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.254637 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-oauth-serving-cert\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.254827 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.254689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc7c2\" (UniqueName: \"kubernetes.io/projected/7576a070-9722-44b8-8331-63f928ed6b68-kube-api-access-cc7c2\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.254827 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.254766 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-console-config\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.254970 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.254836 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7576a070-9722-44b8-8331-63f928ed6b68-console-oauth-config\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.254970 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.254877 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-trusted-ca-bundle\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.298550 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.298505 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:02.355721 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.355677 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-trusted-ca-bundle\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.355911 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.355731 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-service-ca\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.355911 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.355764 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7576a070-9722-44b8-8331-63f928ed6b68-console-serving-cert\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.355911 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.355791 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-oauth-serving-cert\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.355911 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.355813 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7c2\" (UniqueName: \"kubernetes.io/projected/7576a070-9722-44b8-8331-63f928ed6b68-kube-api-access-cc7c2\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.355911 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.355843 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-console-config\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.355911 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.355901 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7576a070-9722-44b8-8331-63f928ed6b68-console-oauth-config\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.356624 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.356594 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-oauth-serving-cert\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.356762 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.356684 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-service-ca\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.356762 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.356707 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-console-config\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.356961 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.356943 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7576a070-9722-44b8-8331-63f928ed6b68-trusted-ca-bundle\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.358909 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.358887 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7576a070-9722-44b8-8331-63f928ed6b68-console-oauth-config\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.359059 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.359040 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7576a070-9722-44b8-8331-63f928ed6b68-console-serving-cert\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.361198 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.361173 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:02.364265 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.364242 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7c2\" (UniqueName: \"kubernetes.io/projected/7576a070-9722-44b8-8331-63f928ed6b68-kube-api-access-cc7c2\") pod \"console-57485bcffb-lk47g\" (UID: \"7576a070-9722-44b8-8331-63f928ed6b68\") " pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.541767 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.541659 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:02.678388 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:02.678349 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57485bcffb-lk47g"] Jul 09 17:47:02.681743 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:47:02.681710 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7576a070_9722_44b8_8331_63f928ed6b68.slice/crio-0e6f2cf50e9d06d559c2a042515bb5c9a459c49ceba1d7c0cc93b438ce73db9c WatchSource:0}: Error finding container 0e6f2cf50e9d06d559c2a042515bb5c9a459c49ceba1d7c0cc93b438ce73db9c: Status 404 returned error can't find the container with id 0e6f2cf50e9d06d559c2a042515bb5c9a459c49ceba1d7c0cc93b438ce73db9c Jul 09 17:47:03.470579 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:03.470532 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57485bcffb-lk47g" event={"ID":"7576a070-9722-44b8-8331-63f928ed6b68","Type":"ContainerStarted","Data":"5273b046f2022f7f54706cea7a7baa6aa23f2909802f9e6dc3e7ca30fc0b93fc"} Jul 09 17:47:03.470579 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:03.470585 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57485bcffb-lk47g" event={"ID":"7576a070-9722-44b8-8331-63f928ed6b68","Type":"ContainerStarted","Data":"0e6f2cf50e9d06d559c2a042515bb5c9a459c49ceba1d7c0cc93b438ce73db9c"} Jul 09 17:47:03.492136 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:03.492082 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-57485bcffb-lk47g" podStartSLOduration=1.4920643820000001 podStartE2EDuration="1.492064382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:47:03.491915833 +0000 UTC m=+451.194417024" watchObservedRunningTime="2026-07-09 17:47:03.492064382 +0000 UTC m=+451.194565572" Jul 09 17:47:11.385205 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:11.385176 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6b8548bc6d-9ff4t" Jul 09 17:47:12.501346 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:12.501306 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:12.542363 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:12.542326 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:12.542568 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:12.542400 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:12.547649 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:12.547620 2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:12.680493 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:12.680463 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:12.777009 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:12.776920 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:13.515358 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:13.515331 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57485bcffb-lk47g" Jul 09 17:47:13.586369 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:13.586331 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f4fd6db6f-2gg5j"] Jul 09 17:47:14.565796 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:14.565748 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:14.642197 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:14.642160 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:20.891102 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:20.891064 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:20.986650 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:20.986600 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:21.625980 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:21.625933 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:21.726632 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:21.726597 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:32.478600 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:32.478560 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:47:38.607483 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.607432 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f4fd6db6f-2gg5j" podUID="f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" containerName="console" containerID="cri-o://358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98" gracePeriod=15 Jul 09 17:47:38.850958 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.850924 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:47:38.959240 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959203 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-service-ca\") pod \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " Jul 09 17:47:38.959454 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959261 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-serving-cert\") pod \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " Jul 09 17:47:38.959454 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959288 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-trusted-ca-bundle\") pod \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " Jul 09 17:47:38.959454 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959306 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcs2c\" (UniqueName: \"kubernetes.io/projected/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-kube-api-access-vcs2c\") pod \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " Jul 09 17:47:38.959454 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959345 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-oauth-serving-cert\") pod \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " Jul 09 17:47:38.959454 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959450 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-config\") pod \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " Jul 09 17:47:38.959717 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959492 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-oauth-config\") pod \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\" (UID: \"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75\") " Jul 09 17:47:38.959802 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959772 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" (UID: "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:47:38.959859 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959697 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-service-ca" (OuterVolumeSpecName: "service-ca") pod "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" (UID: "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:47:38.959937 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959896 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-config" (OuterVolumeSpecName: "console-config") pod "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" (UID: "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:47:38.959937 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.959899 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" (UID: "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:47:38.961703 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.961669 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" (UID: "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:47:38.961823 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.961753 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" (UID: "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:47:38.961823 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:38.961753 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-kube-api-access-vcs2c" (OuterVolumeSpecName: "kube-api-access-vcs2c") pod "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" (UID: "f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75"). InnerVolumeSpecName "kube-api-access-vcs2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 17:47:39.060987 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.060946 2560 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-oauth-serving-cert\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:47:39.060987 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.060976 2560 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-config\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:47:39.060987 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.060986 2560 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-oauth-config\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:47:39.060987 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.060996 2560 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-service-ca\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:47:39.061273 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.061006 2560 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-console-serving-cert\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:47:39.061273 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.061014 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-trusted-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:47:39.061273 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.061023 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcs2c\" (UniqueName: \"kubernetes.io/projected/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75-kube-api-access-vcs2c\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:47:39.615253 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.615218 2560 generic.go:356] "Generic (PLEG): container finished" podID="f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" containerID="358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98" exitCode=0 Jul 09 17:47:39.615709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.615268 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f4fd6db6f-2gg5j" Jul 09 17:47:39.615709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.615278 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4fd6db6f-2gg5j" event={"ID":"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75","Type":"ContainerDied","Data":"358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98"} Jul 09 17:47:39.615709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.615320 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f4fd6db6f-2gg5j" event={"ID":"f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75","Type":"ContainerDied","Data":"df0aceace60170212fb32cb0454788c3a4bd71863ff82a6e256af4ff8b5325ce"} Jul 09 17:47:39.615709 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.615338 2560 scope.go:117] "RemoveContainer" containerID="358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98" Jul 09 17:47:39.624447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.624400 2560 scope.go:117] "RemoveContainer" containerID="358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98" Jul 09 17:47:39.624778 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:47:39.624757 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98\": container with ID starting with 358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98 not found: ID does not exist" containerID="358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98" Jul 09 17:47:39.624867 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.624788 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98"} err="failed to get container status \"358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98\": rpc error: code = NotFound desc = could not find container \"358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98\": container with ID starting with 358491ae51c32e0e417025667024162c40241dc958e72524a995c64c79a85e98 not found: ID does not exist" Jul 09 17:47:39.650031 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.649996 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f4fd6db6f-2gg5j"] Jul 09 17:47:39.652942 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:39.652900 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f4fd6db6f-2gg5j"] Jul 09 17:47:40.811489 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:40.811452 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" path="/var/lib/kubelet/pods/f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75/volumes" Jul 09 17:47:50.752670 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:47:50.752623 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:48:15.787619 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:48:15.787574 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:48:15.860041 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:48:15.860005 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:48:49.217790 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:48:49.217757 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:48:49.319019 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:48:49.318989 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:10.806522 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:10.806486 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:11.140052 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:11.139961 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:11.264186 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:11.264148 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:11.388759 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:11.388725 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:11.574598 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:11.574561 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:11.718886 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:11.718852 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:11.811752 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:11.811716 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:11.919671 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:11.919575 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:49:12.016385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:49:12.016346 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:30.943243 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.943200 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp"] Jul 09 17:50:30.943692 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.943631 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" containerName="console" Jul 09 17:50:30.943692 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.943646 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" containerName="console" Jul 09 17:50:30.943765 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.943727 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f84b77b2-3bd3-4e77-91d7-bf6b2fe11c75" containerName="console" Jul 09 17:50:30.946756 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.946731 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:30.949206 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.949170 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mspnj\"" Jul 09 17:50:30.949206 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.949203 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 09 17:50:30.949447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.949174 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-654f2-serving-cert\"" Jul 09 17:50:30.949447 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.949174 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-654f2-kube-rbac-proxy-sar-config\"" Jul 09 17:50:30.955252 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.955225 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp"] Jul 09 17:50:30.961951 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:30.961914 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:31.013996 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.013960 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956b5d9c-e403-422a-8c6b-eec583c709ec-proxy-tls\") pod \"switch-graph-654f2-66fc497bb7-ndlcp\" (UID: \"956b5d9c-e403-422a-8c6b-eec583c709ec\") " pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:31.014200 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.014015 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956b5d9c-e403-422a-8c6b-eec583c709ec-openshift-service-ca-bundle\") pod \"switch-graph-654f2-66fc497bb7-ndlcp\" (UID: \"956b5d9c-e403-422a-8c6b-eec583c709ec\") " pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:31.088916 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.088878 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:31.115385 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.115335 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956b5d9c-e403-422a-8c6b-eec583c709ec-proxy-tls\") pod \"switch-graph-654f2-66fc497bb7-ndlcp\" (UID: \"956b5d9c-e403-422a-8c6b-eec583c709ec\") " pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:31.115591 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.115395 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956b5d9c-e403-422a-8c6b-eec583c709ec-openshift-service-ca-bundle\") pod \"switch-graph-654f2-66fc497bb7-ndlcp\" (UID: \"956b5d9c-e403-422a-8c6b-eec583c709ec\") " pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:31.116111 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.116079 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956b5d9c-e403-422a-8c6b-eec583c709ec-openshift-service-ca-bundle\") pod \"switch-graph-654f2-66fc497bb7-ndlcp\" (UID: \"956b5d9c-e403-422a-8c6b-eec583c709ec\") " pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:31.117924 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.117901 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956b5d9c-e403-422a-8c6b-eec583c709ec-proxy-tls\") pod \"switch-graph-654f2-66fc497bb7-ndlcp\" (UID: \"956b5d9c-e403-422a-8c6b-eec583c709ec\") " pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:31.259551 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.259510 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:31.395786 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.395750 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp"] Jul 09 17:50:31.399263 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:50:31.399233 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod956b5d9c_e403_422a_8c6b_eec583c709ec.slice/crio-4815c4901b2fdc599531a97ad666c690c952c5d7dd8086ef1c5a2421ab68e16c WatchSource:0}: Error finding container 4815c4901b2fdc599531a97ad666c690c952c5d7dd8086ef1c5a2421ab68e16c: Status 404 returned error can't find the container with id 4815c4901b2fdc599531a97ad666c690c952c5d7dd8086ef1c5a2421ab68e16c Jul 09 17:50:31.401318 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:31.401301 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 17:50:32.269297 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:32.269256 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" event={"ID":"956b5d9c-e403-422a-8c6b-eec583c709ec","Type":"ContainerStarted","Data":"4815c4901b2fdc599531a97ad666c690c952c5d7dd8086ef1c5a2421ab68e16c"} Jul 09 17:50:34.279595 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:34.279561 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" event={"ID":"956b5d9c-e403-422a-8c6b-eec583c709ec","Type":"ContainerStarted","Data":"dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2"} Jul 09 17:50:34.280013 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:34.279676 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:34.295401 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:34.295169 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" podStartSLOduration=1.9062795160000001 podStartE2EDuration="4.295147393s" totalImagesPullingTime="2.388867877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:50:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:50:34.295050154 +0000 UTC m=+661.997551384" watchObservedRunningTime="2026-07-09 17:50:34.295147393 +0000 UTC m=+661.997648580" Jul 09 17:50:40.289156 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:40.289127 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:50:41.047905 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:41.047869 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp"] Jul 09 17:50:41.048106 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:41.048083 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" containerID="cri-o://dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2" gracePeriod=30 Jul 09 17:50:41.065791 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:41.065750 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:41.168133 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:41.168089 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:41.273633 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:41.273600 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:41.369105 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:41.369001 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:41.449521 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:41.449490 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:41.974277 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:41.974241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:42.074690 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:42.074652 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:50:45.287965 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:45.287910 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:50:50.288178 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:50.288138 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:50:55.287850 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:55.287810 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:50:55.288321 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:50:55.287950 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:51:00.288156 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:00.288115 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:51:05.288035 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:05.287996 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:51:10.288412 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:10.288363 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:51:11.197289 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.197264 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:51:11.343507 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.343378 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956b5d9c-e403-422a-8c6b-eec583c709ec-proxy-tls\") pod \"956b5d9c-e403-422a-8c6b-eec583c709ec\" (UID: \"956b5d9c-e403-422a-8c6b-eec583c709ec\") " Jul 09 17:51:11.343507 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.343455 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956b5d9c-e403-422a-8c6b-eec583c709ec-openshift-service-ca-bundle\") pod \"956b5d9c-e403-422a-8c6b-eec583c709ec\" (UID: \"956b5d9c-e403-422a-8c6b-eec583c709ec\") " Jul 09 17:51:11.343904 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.343840 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956b5d9c-e403-422a-8c6b-eec583c709ec-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "956b5d9c-e403-422a-8c6b-eec583c709ec" (UID: "956b5d9c-e403-422a-8c6b-eec583c709ec"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:51:11.345712 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.345688 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/956b5d9c-e403-422a-8c6b-eec583c709ec-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "956b5d9c-e403-422a-8c6b-eec583c709ec" (UID: "956b5d9c-e403-422a-8c6b-eec583c709ec"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:51:11.427121 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.425998 2560 generic.go:356] "Generic (PLEG): container finished" podID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerID="dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2" exitCode=0 Jul 09 17:51:11.427121 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.426057 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" event={"ID":"956b5d9c-e403-422a-8c6b-eec583c709ec","Type":"ContainerDied","Data":"dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2"} Jul 09 17:51:11.427121 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.426155 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" event={"ID":"956b5d9c-e403-422a-8c6b-eec583c709ec","Type":"ContainerDied","Data":"4815c4901b2fdc599531a97ad666c690c952c5d7dd8086ef1c5a2421ab68e16c"} Jul 09 17:51:11.427121 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.426196 2560 scope.go:117] "RemoveContainer" containerID="dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2" Jul 09 17:51:11.427121 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.426440 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp" Jul 09 17:51:11.436218 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.436196 2560 scope.go:117] "RemoveContainer" containerID="dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2" Jul 09 17:51:11.436598 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:51:11.436563 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2\": container with ID starting with dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2 not found: ID does not exist" containerID="dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2" Jul 09 17:51:11.436656 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.436596 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2"} err="failed to get container status \"dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2\": rpc error: code = NotFound desc = could not find container \"dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2\": container with ID starting with dd51154c87088941696f4380f7889652fcb24a912ae6fa237ba82cd43325abe2 not found: ID does not exist" Jul 09 17:51:11.444324 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.444301 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/956b5d9c-e403-422a-8c6b-eec583c709ec-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:51:11.444324 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.444327 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/956b5d9c-e403-422a-8c6b-eec583c709ec-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:51:11.450545 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.450515 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp"] Jul 09 17:51:11.456364 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:11.456333 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-654f2-66fc497bb7-ndlcp"] Jul 09 17:51:12.812869 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:12.812831 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" path="/var/lib/kubelet/pods/956b5d9c-e403-422a-8c6b-eec583c709ec/volumes" Jul 09 17:51:30.916313 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.916273 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-659948ff55-77x5x"] Jul 09 17:51:30.916781 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.916584 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" Jul 09 17:51:30.916781 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.916595 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" Jul 09 17:51:30.916781 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.916645 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="956b5d9c-e403-422a-8c6b-eec583c709ec" containerName="switch-graph-654f2" Jul 09 17:51:30.919497 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.919479 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:30.922765 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.922736 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mspnj\"" Jul 09 17:51:30.922926 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.922736 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Jul 09 17:51:30.922926 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.922884 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Jul 09 17:51:30.923286 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.923257 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 09 17:51:30.931522 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.931494 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-659948ff55-77x5x"] Jul 09 17:51:30.932449 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:30.932407 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:51:31.003094 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.003056 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls\") pod \"model-chainer-659948ff55-77x5x\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:31.003286 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.003174 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ec6369-4d1d-4d65-b38f-dec00dd62844-openshift-service-ca-bundle\") pod \"model-chainer-659948ff55-77x5x\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:31.104404 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.104369 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ec6369-4d1d-4d65-b38f-dec00dd62844-openshift-service-ca-bundle\") pod \"model-chainer-659948ff55-77x5x\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:31.104600 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.104448 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls\") pod \"model-chainer-659948ff55-77x5x\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:31.104600 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:51:31.104580 2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Jul 09 17:51:31.104671 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:51:31.104656 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls podName:c0ec6369-4d1d-4d65-b38f-dec00dd62844 nodeName:}" failed. No retries permitted until 2026-07-09 17:51:31.604636642 +0000 UTC m=+719.307137810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls") pod "model-chainer-659948ff55-77x5x" (UID: "c0ec6369-4d1d-4d65-b38f-dec00dd62844") : secret "model-chainer-serving-cert" not found Jul 09 17:51:31.105080 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.105058 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ec6369-4d1d-4d65-b38f-dec00dd62844-openshift-service-ca-bundle\") pod \"model-chainer-659948ff55-77x5x\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:31.608743 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.608699 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls\") pod \"model-chainer-659948ff55-77x5x\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:31.611174 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.611149 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls\") pod \"model-chainer-659948ff55-77x5x\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:31.831460 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.831400 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:31.966182 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:31.966149 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-659948ff55-77x5x"] Jul 09 17:51:31.969406 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:51:31.969370 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ec6369_4d1d_4d65_b38f_dec00dd62844.slice/crio-0799c10ccecfc4de465241db38b0929e7addcdfffa9b14ca3c8c5cb1bd72efd3 WatchSource:0}: Error finding container 0799c10ccecfc4de465241db38b0929e7addcdfffa9b14ca3c8c5cb1bd72efd3: Status 404 returned error can't find the container with id 0799c10ccecfc4de465241db38b0929e7addcdfffa9b14ca3c8c5cb1bd72efd3 Jul 09 17:51:32.506078 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:32.506046 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" event={"ID":"c0ec6369-4d1d-4d65-b38f-dec00dd62844","Type":"ContainerStarted","Data":"b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0"} Jul 09 17:51:32.506270 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:32.506097 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" event={"ID":"c0ec6369-4d1d-4d65-b38f-dec00dd62844","Type":"ContainerStarted","Data":"0799c10ccecfc4de465241db38b0929e7addcdfffa9b14ca3c8c5cb1bd72efd3"} Jul 09 17:51:32.506270 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:32.506179 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:32.523390 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:32.523333 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" podStartSLOduration=2.523316481 podStartE2EDuration="2.523316481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:51:32.521966593 +0000 UTC m=+720.224467783" watchObservedRunningTime="2026-07-09 17:51:32.523316481 +0000 UTC m=+720.225817671" Jul 09 17:51:32.627602 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:32.627568 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:51:38.515678 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:38.515646 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:41.005393 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.005354 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-659948ff55-77x5x"] Jul 09 17:51:41.005843 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.005598 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" containerID="cri-o://b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0" gracePeriod=30 Jul 09 17:51:41.033223 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.033178 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:51:41.145856 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.145810 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:51:41.280564 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.280455 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:51:41.486854 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.486813 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:51:41.543815 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.543693 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k"] Jul 09 17:51:41.548632 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.548606 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:41.550672 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.550644 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-785cd-serving-cert\"" Jul 09 17:51:41.550829 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.550696 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-785cd-kube-rbac-proxy-sar-config\"" Jul 09 17:51:41.561373 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.561344 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k"] Jul 09 17:51:41.586076 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.586031 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36ce38a9-32eb-4647-b155-e985518e6d07-proxy-tls\") pod \"switch-graph-785cd-7d89ff7948-6lr7k\" (UID: \"36ce38a9-32eb-4647-b155-e985518e6d07\") " pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:41.586241 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.586148 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ce38a9-32eb-4647-b155-e985518e6d07-openshift-service-ca-bundle\") pod \"switch-graph-785cd-7d89ff7948-6lr7k\" (UID: \"36ce38a9-32eb-4647-b155-e985518e6d07\") " pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:41.597803 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.597767 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:51:41.686833 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.686797 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36ce38a9-32eb-4647-b155-e985518e6d07-proxy-tls\") pod \"switch-graph-785cd-7d89ff7948-6lr7k\" (UID: \"36ce38a9-32eb-4647-b155-e985518e6d07\") " pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:41.687042 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.686874 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ce38a9-32eb-4647-b155-e985518e6d07-openshift-service-ca-bundle\") pod \"switch-graph-785cd-7d89ff7948-6lr7k\" (UID: \"36ce38a9-32eb-4647-b155-e985518e6d07\") " pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:41.687624 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.687602 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ce38a9-32eb-4647-b155-e985518e6d07-openshift-service-ca-bundle\") pod \"switch-graph-785cd-7d89ff7948-6lr7k\" (UID: \"36ce38a9-32eb-4647-b155-e985518e6d07\") " pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:41.689690 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.689665 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36ce38a9-32eb-4647-b155-e985518e6d07-proxy-tls\") pod \"switch-graph-785cd-7d89ff7948-6lr7k\" (UID: \"36ce38a9-32eb-4647-b155-e985518e6d07\") " pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:41.718732 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.718692 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:51:41.861583 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:41.861488 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:42.004780 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:42.004746 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k"] Jul 09 17:51:42.008068 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:51:42.008033 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ce38a9_32eb_4647_b155_e985518e6d07.slice/crio-e6e99c30c7f327755e9b0f23eecbd1f6d063dfcf3069d6bd56ea00bb923e9c9a WatchSource:0}: Error finding container e6e99c30c7f327755e9b0f23eecbd1f6d063dfcf3069d6bd56ea00bb923e9c9a: Status 404 returned error can't find the container with id e6e99c30c7f327755e9b0f23eecbd1f6d063dfcf3069d6bd56ea00bb923e9c9a Jul 09 17:51:42.547280 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:42.547240 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" event={"ID":"36ce38a9-32eb-4647-b155-e985518e6d07","Type":"ContainerStarted","Data":"ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166"} Jul 09 17:51:42.547496 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:42.547300 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" event={"ID":"36ce38a9-32eb-4647-b155-e985518e6d07","Type":"ContainerStarted","Data":"e6e99c30c7f327755e9b0f23eecbd1f6d063dfcf3069d6bd56ea00bb923e9c9a"} Jul 09 17:51:42.547496 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:42.547369 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:42.567941 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:42.567883 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" podStartSLOduration=1.567865815 podStartE2EDuration="1.567865815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:51:42.567755299 +0000 UTC m=+730.270256490" watchObservedRunningTime="2026-07-09 17:51:42.567865815 +0000 UTC m=+730.270367005" Jul 09 17:51:43.514691 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:43.514653 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:51:48.514486 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:48.514444 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:51:48.556629 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:48.556597 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 17:51:53.514269 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:53.514224 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:51:53.514751 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:53.514349 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:51:58.514634 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:51:58.514592 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:52:03.514168 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:03.514130 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:52:08.513960 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:08.513910 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 17:52:11.206104 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.206076 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:52:11.340342 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.340246 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls\") pod \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " Jul 09 17:52:11.340342 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.340306 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ec6369-4d1d-4d65-b38f-dec00dd62844-openshift-service-ca-bundle\") pod \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\" (UID: \"c0ec6369-4d1d-4d65-b38f-dec00dd62844\") " Jul 09 17:52:11.340775 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.340748 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ec6369-4d1d-4d65-b38f-dec00dd62844-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "c0ec6369-4d1d-4d65-b38f-dec00dd62844" (UID: "c0ec6369-4d1d-4d65-b38f-dec00dd62844"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 17:52:11.342449 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.342414 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0ec6369-4d1d-4d65-b38f-dec00dd62844" (UID: "c0ec6369-4d1d-4d65-b38f-dec00dd62844"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 17:52:11.441767 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.441722 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0ec6369-4d1d-4d65-b38f-dec00dd62844-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:52:11.441767 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.441757 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0ec6369-4d1d-4d65-b38f-dec00dd62844-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 17:52:11.662979 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.662889 2560 generic.go:356] "Generic (PLEG): container finished" podID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerID="b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0" exitCode=0 Jul 09 17:52:11.662979 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.662931 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" Jul 09 17:52:11.663156 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.662979 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" event={"ID":"c0ec6369-4d1d-4d65-b38f-dec00dd62844","Type":"ContainerDied","Data":"b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0"} Jul 09 17:52:11.663156 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.663031 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-659948ff55-77x5x" event={"ID":"c0ec6369-4d1d-4d65-b38f-dec00dd62844","Type":"ContainerDied","Data":"0799c10ccecfc4de465241db38b0929e7addcdfffa9b14ca3c8c5cb1bd72efd3"} Jul 09 17:52:11.663156 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.663056 2560 scope.go:117] "RemoveContainer" containerID="b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0" Jul 09 17:52:11.675130 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.675111 2560 scope.go:117] "RemoveContainer" containerID="b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0" Jul 09 17:52:11.675481 ip-10-0-141-169 kubenswrapper[2560]: E0709 17:52:11.675460 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0\": container with ID starting with b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0 not found: ID does not exist" containerID="b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0" Jul 09 17:52:11.675559 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.675495 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0"} err="failed to get container status \"b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0\": rpc error: code = NotFound desc = could not find container \"b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0\": container with ID starting with b94ad759fae9aad303276f988979ad6c6cdffc6af9753a7f5074f96eb86015c0 not found: ID does not exist" Jul 09 17:52:11.687634 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.687598 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-659948ff55-77x5x"] Jul 09 17:52:11.693486 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:11.693458 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-659948ff55-77x5x"] Jul 09 17:52:12.811889 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:12.811848 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" path="/var/lib/kubelet/pods/c0ec6369-4d1d-4d65-b38f-dec00dd62844/volumes" Jul 09 17:52:13.045694 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:13.045653 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:52:41.258148 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.258109 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8"] Jul 09 17:52:41.258649 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.258396 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" Jul 09 17:52:41.258649 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.258407 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" Jul 09 17:52:41.258649 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.258486 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ec6369-4d1d-4d65-b38f-dec00dd62844" containerName="model-chainer" Jul 09 17:52:41.262759 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.262724 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.265913 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.265890 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-71ac7-serving-cert\"" Jul 09 17:52:41.267221 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.267195 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-71ac7-kube-rbac-proxy-sar-config\"" Jul 09 17:52:41.267463 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.267443 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:52:41.274212 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.274183 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8"] Jul 09 17:52:41.363230 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.363192 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13710333-bcd3-4c52-8faf-704c819476ff-proxy-tls\") pod \"sequence-graph-71ac7-5968865f6c-x2nj8\" (UID: \"13710333-bcd3-4c52-8faf-704c819476ff\") " pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.363407 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.363268 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13710333-bcd3-4c52-8faf-704c819476ff-openshift-service-ca-bundle\") pod \"sequence-graph-71ac7-5968865f6c-x2nj8\" (UID: \"13710333-bcd3-4c52-8faf-704c819476ff\") " pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.464466 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.464394 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13710333-bcd3-4c52-8faf-704c819476ff-openshift-service-ca-bundle\") pod \"sequence-graph-71ac7-5968865f6c-x2nj8\" (UID: \"13710333-bcd3-4c52-8faf-704c819476ff\") " pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.464671 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.464499 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13710333-bcd3-4c52-8faf-704c819476ff-proxy-tls\") pod \"sequence-graph-71ac7-5968865f6c-x2nj8\" (UID: \"13710333-bcd3-4c52-8faf-704c819476ff\") " pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.465069 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.465045 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13710333-bcd3-4c52-8faf-704c819476ff-openshift-service-ca-bundle\") pod \"sequence-graph-71ac7-5968865f6c-x2nj8\" (UID: \"13710333-bcd3-4c52-8faf-704c819476ff\") " pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.467050 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.467024 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13710333-bcd3-4c52-8faf-704c819476ff-proxy-tls\") pod \"sequence-graph-71ac7-5968865f6c-x2nj8\" (UID: \"13710333-bcd3-4c52-8faf-704c819476ff\") " pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.528150 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.528054 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:52:41.574074 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.574038 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.590524 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.590485 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:52:41.716042 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.716001 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8"] Jul 09 17:52:41.719240 ip-10-0-141-169 kubenswrapper[2560]: W0709 17:52:41.719208 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13710333_bcd3_4c52_8faf_704c819476ff.slice/crio-3eca93a2463b665f07cb5c1d323b87faab9485dc23ef33b0c946a96e121667c9 WatchSource:0}: Error finding container 3eca93a2463b665f07cb5c1d323b87faab9485dc23ef33b0c946a96e121667c9: Status 404 returned error can't find the container with id 3eca93a2463b665f07cb5c1d323b87faab9485dc23ef33b0c946a96e121667c9 Jul 09 17:52:41.782742 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.782662 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" event={"ID":"13710333-bcd3-4c52-8faf-704c819476ff","Type":"ContainerStarted","Data":"f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e"} Jul 09 17:52:41.782742 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.782717 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" event={"ID":"13710333-bcd3-4c52-8faf-704c819476ff","Type":"ContainerStarted","Data":"3eca93a2463b665f07cb5c1d323b87faab9485dc23ef33b0c946a96e121667c9"} Jul 09 17:52:41.782964 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.782814 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:52:41.800220 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:41.800156 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" podStartSLOduration=0.800136827 podStartE2EDuration="800.136827ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:52:41.798833706 +0000 UTC m=+789.501334912" watchObservedRunningTime="2026-07-09 17:52:41.800136827 +0000 UTC m=+789.502638061" Jul 09 17:52:47.792659 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:52:47.792627 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 17:53:01.758260 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:53:01.758161 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:56:23.086934 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:56:23.086838 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:57:15.526966 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:57:15.526927 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:57:27.166322 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:57:27.166231 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:58:23.905618 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:58:23.905538 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:59:55.855474 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:55.855437 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k"] Jul 09 17:59:55.856009 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:55.855664 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" containerID="cri-o://ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166" gracePeriod=30 Jul 09 17:59:55.920558 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:55.920516 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:59:56.048144 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:56.048106 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:59:56.132695 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:56.132600 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:59:56.229848 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:56.229814 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:59:56.296504 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:56.296466 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:59:56.398253 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:56.398159 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:59:56.499938 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:56.499891 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 17:59:58.555529 ip-10-0-141-169 kubenswrapper[2560]: I0709 17:59:58.555484 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:00:03.555764 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:03.555723 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:00:08.555437 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:08.555371 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:00:08.555892 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:08.555503 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 18:00:13.554833 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:13.554792 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:00:18.555750 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:18.555711 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:00:23.554903 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:23.554862 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:00:26.006850 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.006823 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 18:00:26.115670 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.115566 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36ce38a9-32eb-4647-b155-e985518e6d07-proxy-tls\") pod \"36ce38a9-32eb-4647-b155-e985518e6d07\" (UID: \"36ce38a9-32eb-4647-b155-e985518e6d07\") " Jul 09 18:00:26.115670 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.115614 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ce38a9-32eb-4647-b155-e985518e6d07-openshift-service-ca-bundle\") pod \"36ce38a9-32eb-4647-b155-e985518e6d07\" (UID: \"36ce38a9-32eb-4647-b155-e985518e6d07\") " Jul 09 18:00:26.116016 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.115993 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ce38a9-32eb-4647-b155-e985518e6d07-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "36ce38a9-32eb-4647-b155-e985518e6d07" (UID: "36ce38a9-32eb-4647-b155-e985518e6d07"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:00:26.117968 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.117937 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ce38a9-32eb-4647-b155-e985518e6d07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36ce38a9-32eb-4647-b155-e985518e6d07" (UID: "36ce38a9-32eb-4647-b155-e985518e6d07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:00:26.216583 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.216537 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36ce38a9-32eb-4647-b155-e985518e6d07-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:00:26.216583 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.216577 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36ce38a9-32eb-4647-b155-e985518e6d07-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:00:26.680654 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.680616 2560 generic.go:356] "Generic (PLEG): container finished" podID="36ce38a9-32eb-4647-b155-e985518e6d07" containerID="ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166" exitCode=0 Jul 09 18:00:26.680654 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.680658 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" Jul 09 18:00:26.680890 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.680694 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" event={"ID":"36ce38a9-32eb-4647-b155-e985518e6d07","Type":"ContainerDied","Data":"ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166"} Jul 09 18:00:26.680890 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.680775 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k" event={"ID":"36ce38a9-32eb-4647-b155-e985518e6d07","Type":"ContainerDied","Data":"e6e99c30c7f327755e9b0f23eecbd1f6d063dfcf3069d6bd56ea00bb923e9c9a"} Jul 09 18:00:26.680890 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.680797 2560 scope.go:117] "RemoveContainer" containerID="ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166" Jul 09 18:00:26.689330 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.689312 2560 scope.go:117] "RemoveContainer" containerID="ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166" Jul 09 18:00:26.689694 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:00:26.689672 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166\": container with ID starting with ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166 not found: ID does not exist" containerID="ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166" Jul 09 18:00:26.689789 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.689702 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166"} err="failed to get container status \"ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166\": rpc error: code = NotFound desc = could not find container \"ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166\": container with ID starting with ebab3f907e06196acea15aae1e7c59c6122212970d6328b4b750434071469166 not found: ID does not exist" Jul 09 18:00:26.700571 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.700537 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k"] Jul 09 18:00:26.705547 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.705515 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-785cd-7d89ff7948-6lr7k"] Jul 09 18:00:26.812275 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:26.812237 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" path="/var/lib/kubelet/pods/36ce38a9-32eb-4647-b155-e985518e6d07/volumes" Jul 09 18:00:55.958511 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:55.958475 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8"] Jul 09 18:00:55.958948 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:55.958741 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" containerID="cri-o://f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e" gracePeriod=30 Jul 09 18:00:55.986006 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:55.985968 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:00:56.135012 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.134965 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:00:56.137172 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.137139 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj"] Jul 09 18:00:56.137778 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.137588 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" Jul 09 18:00:56.137778 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.137611 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" Jul 09 18:00:56.137778 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.137698 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ce38a9-32eb-4647-b155-e985518e6d07" containerName="switch-graph-785cd" Jul 09 18:00:56.142016 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.141991 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj"] Jul 09 18:00:56.142147 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.142123 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:56.144875 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.144619 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-5bd2e-serving-cert\"" Jul 09 18:00:56.144875 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.144724 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-5bd2e-kube-rbac-proxy-sar-config\"" Jul 09 18:00:56.228538 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.228387 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:00:56.232705 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.232677 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-openshift-service-ca-bundle\") pod \"ensemble-graph-5bd2e-845cdb685d-fmrbj\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:56.232891 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.232736 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls\") pod \"ensemble-graph-5bd2e-845cdb685d-fmrbj\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:56.322726 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.322686 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:00:56.333917 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.333878 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-openshift-service-ca-bundle\") pod \"ensemble-graph-5bd2e-845cdb685d-fmrbj\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:56.334223 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.334199 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls\") pod \"ensemble-graph-5bd2e-845cdb685d-fmrbj\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:56.334355 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:00:56.334331 2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-5bd2e-serving-cert: secret "ensemble-graph-5bd2e-serving-cert" not found Jul 09 18:00:56.334463 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:00:56.334447 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls podName:b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4 nodeName:}" failed. No retries permitted until 2026-07-09 18:00:56.834396047 +0000 UTC m=+1284.536897219 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls") pod "ensemble-graph-5bd2e-845cdb685d-fmrbj" (UID: "b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4") : secret "ensemble-graph-5bd2e-serving-cert" not found Jul 09 18:00:56.334919 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.334900 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-openshift-service-ca-bundle\") pod \"ensemble-graph-5bd2e-845cdb685d-fmrbj\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:56.420806 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.420770 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:00:56.521973 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.521873 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:00:56.622332 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.622300 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:00:56.837575 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.837473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls\") pod \"ensemble-graph-5bd2e-845cdb685d-fmrbj\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:56.840158 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:56.840116 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls\") pod \"ensemble-graph-5bd2e-845cdb685d-fmrbj\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:57.056870 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:57.056829 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:57.198226 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:57.198190 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj"] Jul 09 18:00:57.201576 ip-10-0-141-169 kubenswrapper[2560]: W0709 18:00:57.201538 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46fd1cc_3acd_47c6_a5b4_7cbeb4caaed4.slice/crio-7ff0c75314d2fc4b30d5e19555f7862ad7f8a8518e6d280cc1025ebda8778294 WatchSource:0}: Error finding container 7ff0c75314d2fc4b30d5e19555f7862ad7f8a8518e6d280cc1025ebda8778294: Status 404 returned error can't find the container with id 7ff0c75314d2fc4b30d5e19555f7862ad7f8a8518e6d280cc1025ebda8778294 Jul 09 18:00:57.791312 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:57.791269 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:00:57.806950 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:57.806915 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" event={"ID":"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4","Type":"ContainerStarted","Data":"b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af"} Jul 09 18:00:57.807129 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:57.806965 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" event={"ID":"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4","Type":"ContainerStarted","Data":"7ff0c75314d2fc4b30d5e19555f7862ad7f8a8518e6d280cc1025ebda8778294"} Jul 09 18:00:57.807129 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:57.807004 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:00:57.825771 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:00:57.825714 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" podStartSLOduration=1.825697507 podStartE2EDuration="1.825697507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:00:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:00:57.824363212 +0000 UTC m=+1285.526864401" watchObservedRunningTime="2026-07-09 18:00:57.825697507 +0000 UTC m=+1285.528198696" Jul 09 18:01:02.790900 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:02.790851 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:03.817263 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:03.817231 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:01:06.186263 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:06.186160 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj"] Jul 09 18:01:06.186717 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:06.186350 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" containerID="cri-o://b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af" gracePeriod=30 Jul 09 18:01:06.206195 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:06.206136 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:01:06.317526 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:06.317478 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:01:06.397279 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:06.397241 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:01:06.506285 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:06.506244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:01:06.600201 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:06.600166 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:01:06.700821 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:06.700778 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:01:07.791361 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:07.791318 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:07.791796 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:07.791457 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 18:01:08.815658 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:08.815621 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:12.790990 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:12.790946 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:13.815938 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:13.815881 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:17.791089 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:17.791042 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:18.815435 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:18.815379 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:18.815873 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:18.815510 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:01:22.791736 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:22.791689 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:23.815763 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:23.815720 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:26.114709 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.114684 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 18:01:26.195263 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.195232 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13710333-bcd3-4c52-8faf-704c819476ff-openshift-service-ca-bundle\") pod \"13710333-bcd3-4c52-8faf-704c819476ff\" (UID: \"13710333-bcd3-4c52-8faf-704c819476ff\") " Jul 09 18:01:26.195488 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.195283 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13710333-bcd3-4c52-8faf-704c819476ff-proxy-tls\") pod \"13710333-bcd3-4c52-8faf-704c819476ff\" (UID: \"13710333-bcd3-4c52-8faf-704c819476ff\") " Jul 09 18:01:26.195641 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.195615 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13710333-bcd3-4c52-8faf-704c819476ff-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "13710333-bcd3-4c52-8faf-704c819476ff" (UID: "13710333-bcd3-4c52-8faf-704c819476ff"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:01:26.197624 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.197578 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13710333-bcd3-4c52-8faf-704c819476ff-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13710333-bcd3-4c52-8faf-704c819476ff" (UID: "13710333-bcd3-4c52-8faf-704c819476ff"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:01:26.296756 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.296662 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/13710333-bcd3-4c52-8faf-704c819476ff-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:01:26.296756 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.296691 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13710333-bcd3-4c52-8faf-704c819476ff-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:01:26.927339 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.927303 2560 generic.go:356] "Generic (PLEG): container finished" podID="13710333-bcd3-4c52-8faf-704c819476ff" containerID="f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e" exitCode=0 Jul 09 18:01:26.927537 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.927358 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" event={"ID":"13710333-bcd3-4c52-8faf-704c819476ff","Type":"ContainerDied","Data":"f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e"} Jul 09 18:01:26.927537 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.927397 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" event={"ID":"13710333-bcd3-4c52-8faf-704c819476ff","Type":"ContainerDied","Data":"3eca93a2463b665f07cb5c1d323b87faab9485dc23ef33b0c946a96e121667c9"} Jul 09 18:01:26.927537 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.927359 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8" Jul 09 18:01:26.927537 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.927440 2560 scope.go:117] "RemoveContainer" containerID="f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e" Jul 09 18:01:26.938259 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.938237 2560 scope.go:117] "RemoveContainer" containerID="f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e" Jul 09 18:01:26.938576 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:01:26.938557 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e\": container with ID starting with f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e not found: ID does not exist" containerID="f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e" Jul 09 18:01:26.938626 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.938585 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e"} err="failed to get container status \"f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e\": rpc error: code = NotFound desc = could not find container \"f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e\": container with ID starting with f93684b3f25b874705dabd4c6aefb67637082977127ded03b4e4a4adf57edf3e not found: ID does not exist" Jul 09 18:01:26.948282 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.948247 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8"] Jul 09 18:01:26.954355 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:26.954323 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-71ac7-5968865f6c-x2nj8"] Jul 09 18:01:28.811989 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:28.811949 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13710333-bcd3-4c52-8faf-704c819476ff" path="/var/lib/kubelet/pods/13710333-bcd3-4c52-8faf-704c819476ff/volumes" Jul 09 18:01:28.816047 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:28.816013 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:33.815241 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:33.815198 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:01:36.842217 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.842191 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:01:36.969974 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.969875 2560 generic.go:356] "Generic (PLEG): container finished" podID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerID="b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af" exitCode=0 Jul 09 18:01:36.969974 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.969924 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" Jul 09 18:01:36.969974 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.969963 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" event={"ID":"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4","Type":"ContainerDied","Data":"b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af"} Jul 09 18:01:36.970215 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.970023 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj" event={"ID":"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4","Type":"ContainerDied","Data":"7ff0c75314d2fc4b30d5e19555f7862ad7f8a8518e6d280cc1025ebda8778294"} Jul 09 18:01:36.970215 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.970053 2560 scope.go:117] "RemoveContainer" containerID="b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af" Jul 09 18:01:36.977917 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.977897 2560 scope.go:117] "RemoveContainer" containerID="b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af" Jul 09 18:01:36.978200 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:01:36.978180 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af\": container with ID starting with b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af not found: ID does not exist" containerID="b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af" Jul 09 18:01:36.978247 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.978210 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af"} err="failed to get container status \"b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af\": rpc error: code = NotFound desc = could not find container \"b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af\": container with ID starting with b27ddec58c5a5c6a231ba94e4f929e3bb03e46de7f638cfefaa8c8b61b93c0af not found: ID does not exist" Jul 09 18:01:36.984526 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.984506 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls\") pod \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " Jul 09 18:01:36.984604 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.984591 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-openshift-service-ca-bundle\") pod \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\" (UID: \"b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4\") " Jul 09 18:01:36.984958 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.984939 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" (UID: "b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:01:36.986657 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:36.986639 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" (UID: "b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:01:37.085853 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:37.085816 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:01:37.085853 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:37.085848 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:01:37.290811 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:37.290777 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj"] Jul 09 18:01:37.293057 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:37.293032 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5bd2e-845cdb685d-fmrbj"] Jul 09 18:01:38.811242 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:38.811203 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" path="/var/lib/kubelet/pods/b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4/volumes" Jul 09 18:01:56.172862 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.172781 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4"] Jul 09 18:01:56.173248 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.173092 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" Jul 09 18:01:56.173248 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.173104 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" Jul 09 18:01:56.173248 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.173118 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" Jul 09 18:01:56.173248 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.173124 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" Jul 09 18:01:56.173248 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.173175 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="13710333-bcd3-4c52-8faf-704c819476ff" containerName="sequence-graph-71ac7" Jul 09 18:01:56.173248 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.173185 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b46fd1cc-3acd-47c6-a5b4-7cbeb4caaed4" containerName="ensemble-graph-5bd2e" Jul 09 18:01:56.177542 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.177517 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:56.179612 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.179587 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-53a2e-serving-cert\"" Jul 09 18:01:56.179612 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.179602 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 09 18:01:56.180164 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.180147 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-53a2e-kube-rbac-proxy-sar-config\"" Jul 09 18:01:56.180249 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.180189 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mspnj\"" Jul 09 18:01:56.192223 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.192194 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4"] Jul 09 18:01:56.194957 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.194927 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:01:56.233103 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.233061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-openshift-service-ca-bundle\") pod \"sequence-graph-53a2e-5dd9575f9b-gddb4\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:56.233323 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.233108 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls\") pod \"sequence-graph-53a2e-5dd9575f9b-gddb4\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:56.333772 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.333728 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-openshift-service-ca-bundle\") pod \"sequence-graph-53a2e-5dd9575f9b-gddb4\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:56.333989 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.333785 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls\") pod \"sequence-graph-53a2e-5dd9575f9b-gddb4\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:56.333989 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:01:56.333925 2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-53a2e-serving-cert: secret "sequence-graph-53a2e-serving-cert" not found Jul 09 18:01:56.334114 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:01:56.334011 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls podName:98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe nodeName:}" failed. No retries permitted until 2026-07-09 18:01:56.833987666 +0000 UTC m=+1344.536488833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls") pod "sequence-graph-53a2e-5dd9575f9b-gddb4" (UID: "98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe") : secret "sequence-graph-53a2e-serving-cert" not found Jul 09 18:01:56.334403 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.334379 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-openshift-service-ca-bundle\") pod \"sequence-graph-53a2e-5dd9575f9b-gddb4\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:56.635312 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.635272 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:01:56.837051 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.837001 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls\") pod \"sequence-graph-53a2e-5dd9575f9b-gddb4\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:56.839428 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:56.839404 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls\") pod \"sequence-graph-53a2e-5dd9575f9b-gddb4\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:57.088034 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:57.087991 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:57.224644 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:57.224610 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4"] Jul 09 18:01:57.228358 ip-10-0-141-169 kubenswrapper[2560]: W0709 18:01:57.228326 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98b9d6c5_d6e7_4d52_b3e2_4a01c9f005fe.slice/crio-9add4bcf19cd267e85276d60ac3c387843a2c8f54b2b0dc5c19fd8f4b20aeeb5 WatchSource:0}: Error finding container 9add4bcf19cd267e85276d60ac3c387843a2c8f54b2b0dc5c19fd8f4b20aeeb5: Status 404 returned error can't find the container with id 9add4bcf19cd267e85276d60ac3c387843a2c8f54b2b0dc5c19fd8f4b20aeeb5 Jul 09 18:01:58.053188 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:58.053155 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" event={"ID":"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe","Type":"ContainerStarted","Data":"ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981"} Jul 09 18:01:58.053375 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:58.053206 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" event={"ID":"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe","Type":"ContainerStarted","Data":"9add4bcf19cd267e85276d60ac3c387843a2c8f54b2b0dc5c19fd8f4b20aeeb5"} Jul 09 18:01:58.053375 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:58.053236 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:01:58.071804 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:01:58.071750 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" podStartSLOduration=2.071730327 podStartE2EDuration="2.071730327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:01:58.070753593 +0000 UTC m=+1345.773254810" watchObservedRunningTime="2026-07-09 18:01:58.071730327 +0000 UTC m=+1345.774231548" Jul 09 18:02:04.063936 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:04.063908 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:02:06.259060 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.259023 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4"] Jul 09 18:02:06.259563 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.259219 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" containerID="cri-o://ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981" gracePeriod=30 Jul 09 18:02:06.286174 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.286139 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:02:06.454729 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.454622 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:02:06.468079 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.468042 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j"] Jul 09 18:02:06.471728 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.471701 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:06.481198 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.481169 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-14ef6-kube-rbac-proxy-sar-config\"" Jul 09 18:02:06.489526 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.489495 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-14ef6-serving-cert\"" Jul 09 18:02:06.508945 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.508912 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j"] Jul 09 18:02:06.509308 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.509264 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacfc064-afd8-4401-9397-02488702a206-openshift-service-ca-bundle\") pod \"ensemble-graph-14ef6-8648bc46f9-f667j\" (UID: \"eacfc064-afd8-4401-9397-02488702a206\") " pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:06.509359 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.509303 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacfc064-afd8-4401-9397-02488702a206-proxy-tls\") pod \"ensemble-graph-14ef6-8648bc46f9-f667j\" (UID: \"eacfc064-afd8-4401-9397-02488702a206\") " pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:06.610086 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.610046 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacfc064-afd8-4401-9397-02488702a206-openshift-service-ca-bundle\") pod \"ensemble-graph-14ef6-8648bc46f9-f667j\" (UID: \"eacfc064-afd8-4401-9397-02488702a206\") " pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:06.610272 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.610122 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacfc064-afd8-4401-9397-02488702a206-proxy-tls\") pod \"ensemble-graph-14ef6-8648bc46f9-f667j\" (UID: \"eacfc064-afd8-4401-9397-02488702a206\") " pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:06.610817 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.610784 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacfc064-afd8-4401-9397-02488702a206-openshift-service-ca-bundle\") pod \"ensemble-graph-14ef6-8648bc46f9-f667j\" (UID: \"eacfc064-afd8-4401-9397-02488702a206\") " pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:06.612803 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.612781 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacfc064-afd8-4401-9397-02488702a206-proxy-tls\") pod \"ensemble-graph-14ef6-8648bc46f9-f667j\" (UID: \"eacfc064-afd8-4401-9397-02488702a206\") " pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:06.709056 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.709017 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:02:06.784396 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.784200 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:06.811913 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.811521 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:02:06.878984 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.878918 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:02:06.932829 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:06.932796 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j"] Jul 09 18:02:06.936474 ip-10-0-141-169 kubenswrapper[2560]: W0709 18:02:06.936435 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeacfc064_afd8_4401_9397_02488702a206.slice/crio-ddbbd7d7db5c60a49ab6bfa377b5182e0a79f4de1d7ee93d72cadaf2a2f6a7da WatchSource:0}: Error finding container ddbbd7d7db5c60a49ab6bfa377b5182e0a79f4de1d7ee93d72cadaf2a2f6a7da: Status 404 returned error can't find the container with id ddbbd7d7db5c60a49ab6bfa377b5182e0a79f4de1d7ee93d72cadaf2a2f6a7da Jul 09 18:02:07.091092 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:07.090974 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:02:07.092472 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:07.092446 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" event={"ID":"eacfc064-afd8-4401-9397-02488702a206","Type":"ContainerStarted","Data":"246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e"} Jul 09 18:02:07.092633 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:07.092497 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" event={"ID":"eacfc064-afd8-4401-9397-02488702a206","Type":"ContainerStarted","Data":"ddbbd7d7db5c60a49ab6bfa377b5182e0a79f4de1d7ee93d72cadaf2a2f6a7da"} Jul 09 18:02:07.092633 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:07.092572 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:07.110795 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:07.110654 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" podStartSLOduration=1.110630549 podStartE2EDuration="1.110630549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:02:07.109900985 +0000 UTC m=+1354.812402174" watchObservedRunningTime="2026-07-09 18:02:07.110630549 +0000 UTC m=+1354.813131741" Jul 09 18:02:09.062545 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:09.062505 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:02:13.102801 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:13.102770 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:02:14.062672 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:14.062628 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:02:19.062816 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:19.062774 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:02:19.063360 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:19.062883 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:02:24.062377 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:24.062333 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:02:24.613337 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:24.613285 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:02:29.062375 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:29.062331 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:02:34.062022 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:34.061976 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:02:36.414002 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:36.413975 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:02:36.562782 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:36.562690 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-openshift-service-ca-bundle\") pod \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " Jul 09 18:02:36.562782 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:36.562732 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls\") pod \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\" (UID: \"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe\") " Jul 09 18:02:36.563097 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:36.563075 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" (UID: "98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:02:36.564879 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:36.564840 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" (UID: "98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:02:36.663787 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:36.663741 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:02:36.663787 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:36.663777 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:02:37.212552 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.212521 2560 generic.go:356] "Generic (PLEG): container finished" podID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerID="ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981" exitCode=0 Jul 09 18:02:37.212741 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.212565 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" Jul 09 18:02:37.212741 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.212585 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" event={"ID":"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe","Type":"ContainerDied","Data":"ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981"} Jul 09 18:02:37.212741 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.212629 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4" event={"ID":"98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe","Type":"ContainerDied","Data":"9add4bcf19cd267e85276d60ac3c387843a2c8f54b2b0dc5c19fd8f4b20aeeb5"} Jul 09 18:02:37.212741 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.212649 2560 scope.go:117] "RemoveContainer" containerID="ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981" Jul 09 18:02:37.221072 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.221052 2560 scope.go:117] "RemoveContainer" containerID="ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981" Jul 09 18:02:37.221402 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:02:37.221379 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981\": container with ID starting with ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981 not found: ID does not exist" containerID="ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981" Jul 09 18:02:37.221568 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.221408 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981"} err="failed to get container status \"ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981\": rpc error: code = NotFound desc = could not find container \"ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981\": container with ID starting with ed4b4063b42e17355905b914b26fc21d0c6aa07914d526dbcfd03b64e9c37981 not found: ID does not exist" Jul 09 18:02:37.227533 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.227500 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4"] Jul 09 18:02:37.231648 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:37.231622 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-53a2e-5dd9575f9b-gddb4"] Jul 09 18:02:38.812036 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:02:38.812000 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" path="/var/lib/kubelet/pods/98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe/volumes" Jul 09 18:03:06.519818 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.519776 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw"] Jul 09 18:03:06.520290 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.520111 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" Jul 09 18:03:06.520290 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.520123 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" Jul 09 18:03:06.520290 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.520170 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98b9d6c5-d6e7-4d52-b3e2-4a01c9f005fe" containerName="sequence-graph-53a2e" Jul 09 18:03:06.523078 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.523058 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:06.525071 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.525049 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-67922-serving-cert\"" Jul 09 18:03:06.525214 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.525081 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-67922-kube-rbac-proxy-sar-config\"" Jul 09 18:03:06.530570 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.530539 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw"] Jul 09 18:03:06.534625 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.534588 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:03:06.613770 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.613729 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d51d90-3774-4d90-8738-3c66b2a1a90b-openshift-service-ca-bundle\") pod \"sequence-graph-67922-65cfc7d557-46nnw\" (UID: \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\") " pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:06.613958 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.613784 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7d51d90-3774-4d90-8738-3c66b2a1a90b-proxy-tls\") pod \"sequence-graph-67922-65cfc7d557-46nnw\" (UID: \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\") " pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:06.714442 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.714368 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d51d90-3774-4d90-8738-3c66b2a1a90b-openshift-service-ca-bundle\") pod \"sequence-graph-67922-65cfc7d557-46nnw\" (UID: \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\") " pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:06.714641 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.714466 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7d51d90-3774-4d90-8738-3c66b2a1a90b-proxy-tls\") pod \"sequence-graph-67922-65cfc7d557-46nnw\" (UID: \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\") " pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:06.715089 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.715057 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d51d90-3774-4d90-8738-3c66b2a1a90b-openshift-service-ca-bundle\") pod \"sequence-graph-67922-65cfc7d557-46nnw\" (UID: \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\") " pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:06.716940 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.716919 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7d51d90-3774-4d90-8738-3c66b2a1a90b-proxy-tls\") pod \"sequence-graph-67922-65cfc7d557-46nnw\" (UID: \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\") " pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:06.837090 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.836978 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:06.898578 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.898536 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:03:06.988002 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:06.987816 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw"] Jul 09 18:03:06.990906 ip-10-0-141-169 kubenswrapper[2560]: W0709 18:03:06.990873 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d51d90_3774_4d90_8738_3c66b2a1a90b.slice/crio-d65c15dfdf5f1e796b3c4c3f74dfded34085a6125d405bfea1b776a9b9ad7770 WatchSource:0}: Error finding container d65c15dfdf5f1e796b3c4c3f74dfded34085a6125d405bfea1b776a9b9ad7770: Status 404 returned error can't find the container with id d65c15dfdf5f1e796b3c4c3f74dfded34085a6125d405bfea1b776a9b9ad7770 Jul 09 18:03:07.336472 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:07.336400 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" event={"ID":"f7d51d90-3774-4d90-8738-3c66b2a1a90b","Type":"ContainerStarted","Data":"a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e"} Jul 09 18:03:07.336679 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:07.336489 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" event={"ID":"f7d51d90-3774-4d90-8738-3c66b2a1a90b","Type":"ContainerStarted","Data":"d65c15dfdf5f1e796b3c4c3f74dfded34085a6125d405bfea1b776a9b9ad7770"} Jul 09 18:03:07.336679 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:07.336517 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:03:07.354289 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:07.354231 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" podStartSLOduration=1.354212605 podStartE2EDuration="1.354212605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:03:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:03:07.353117298 +0000 UTC m=+1415.055618489" watchObservedRunningTime="2026-07-09 18:03:07.354212605 +0000 UTC m=+1415.056713794" Jul 09 18:03:13.345797 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:03:13.345765 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:05:30.369618 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:05:30.369581 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:10:20.933627 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:20.933592 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j"] Jul 09 18:10:20.934223 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:20.933810 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" containerID="cri-o://246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e" gracePeriod=30 Jul 09 18:10:20.991251 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:20.991214 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:10:23.101172 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:23.101121 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:10:23.586802 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:23.586764 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:10:28.101406 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:28.101364 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:10:33.101033 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:33.100990 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:10:33.101490 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:33.101110 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:10:38.101313 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:38.101270 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:10:43.100875 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:43.100830 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:10:48.101399 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:48.101355 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:10:51.093777 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.093748 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:10:51.163245 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.163207 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacfc064-afd8-4401-9397-02488702a206-openshift-service-ca-bundle\") pod \"eacfc064-afd8-4401-9397-02488702a206\" (UID: \"eacfc064-afd8-4401-9397-02488702a206\") " Jul 09 18:10:51.163404 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.163332 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacfc064-afd8-4401-9397-02488702a206-proxy-tls\") pod \"eacfc064-afd8-4401-9397-02488702a206\" (UID: \"eacfc064-afd8-4401-9397-02488702a206\") " Jul 09 18:10:51.163604 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.163584 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eacfc064-afd8-4401-9397-02488702a206-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "eacfc064-afd8-4401-9397-02488702a206" (UID: "eacfc064-afd8-4401-9397-02488702a206"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:10:51.165505 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.165475 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eacfc064-afd8-4401-9397-02488702a206-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eacfc064-afd8-4401-9397-02488702a206" (UID: "eacfc064-afd8-4401-9397-02488702a206"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:10:51.225513 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.225467 2560 generic.go:356] "Generic (PLEG): container finished" podID="eacfc064-afd8-4401-9397-02488702a206" containerID="246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e" exitCode=0 Jul 09 18:10:51.225513 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.225521 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" Jul 09 18:10:51.225759 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.225561 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" event={"ID":"eacfc064-afd8-4401-9397-02488702a206","Type":"ContainerDied","Data":"246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e"} Jul 09 18:10:51.225759 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.225619 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j" event={"ID":"eacfc064-afd8-4401-9397-02488702a206","Type":"ContainerDied","Data":"ddbbd7d7db5c60a49ab6bfa377b5182e0a79f4de1d7ee93d72cadaf2a2f6a7da"} Jul 09 18:10:51.225759 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.225642 2560 scope.go:117] "RemoveContainer" containerID="246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e" Jul 09 18:10:51.240813 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.240790 2560 scope.go:117] "RemoveContainer" containerID="246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e" Jul 09 18:10:51.241165 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:10:51.241137 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e\": container with ID starting with 246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e not found: ID does not exist" containerID="246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e" Jul 09 18:10:51.241272 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.241169 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e"} err="failed to get container status \"246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e\": rpc error: code = NotFound desc = could not find container \"246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e\": container with ID starting with 246e85f159a7ae78b7db6bc865fa2f329e2e511849c84da7e6bde076d1485f3e not found: ID does not exist" Jul 09 18:10:51.249344 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.249307 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j"] Jul 09 18:10:51.252775 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.252749 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-14ef6-8648bc46f9-f667j"] Jul 09 18:10:51.264719 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.264682 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/eacfc064-afd8-4401-9397-02488702a206-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:10:51.264719 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:51.264721 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eacfc064-afd8-4401-9397-02488702a206-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:10:52.811663 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:10:52.811628 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eacfc064-afd8-4401-9397-02488702a206" path="/var/lib/kubelet/pods/eacfc064-afd8-4401-9397-02488702a206/volumes" Jul 09 18:11:21.099346 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.099265 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw"] Jul 09 18:11:21.099785 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.099529 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" containerID="cri-o://a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e" gracePeriod=30 Jul 09 18:11:21.119035 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.118992 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:21.192113 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.192072 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j"] Jul 09 18:11:21.192400 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.192368 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" Jul 09 18:11:21.192460 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.192402 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" Jul 09 18:11:21.192495 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.192477 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="eacfc064-afd8-4401-9397-02488702a206" containerName="ensemble-graph-14ef6" Jul 09 18:11:21.195288 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.195261 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:21.197899 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.197875 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-07956-kube-rbac-proxy-sar-config\"" Jul 09 18:11:21.198129 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.198111 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-07956-serving-cert\"" Jul 09 18:11:21.205914 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.205885 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j"] Jul 09 18:11:21.215204 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.215164 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:21.311435 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.311382 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ebe67c6-85cc-45f1-a559-63329fdf7389-proxy-tls\") pod \"splitter-graph-07956-66ccc97694-fwn8j\" (UID: \"9ebe67c6-85cc-45f1-a559-63329fdf7389\") " pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:21.311620 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.311508 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebe67c6-85cc-45f1-a559-63329fdf7389-openshift-service-ca-bundle\") pod \"splitter-graph-07956-66ccc97694-fwn8j\" (UID: \"9ebe67c6-85cc-45f1-a559-63329fdf7389\") " pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:21.320021 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.319989 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:21.406301 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.406215 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:21.412414 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.412382 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebe67c6-85cc-45f1-a559-63329fdf7389-openshift-service-ca-bundle\") pod \"splitter-graph-07956-66ccc97694-fwn8j\" (UID: \"9ebe67c6-85cc-45f1-a559-63329fdf7389\") " pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:21.412583 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.412473 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ebe67c6-85cc-45f1-a559-63329fdf7389-proxy-tls\") pod \"splitter-graph-07956-66ccc97694-fwn8j\" (UID: \"9ebe67c6-85cc-45f1-a559-63329fdf7389\") " pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:21.413013 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.412993 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebe67c6-85cc-45f1-a559-63329fdf7389-openshift-service-ca-bundle\") pod \"splitter-graph-07956-66ccc97694-fwn8j\" (UID: \"9ebe67c6-85cc-45f1-a559-63329fdf7389\") " pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:21.414827 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.414806 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ebe67c6-85cc-45f1-a559-63329fdf7389-proxy-tls\") pod \"splitter-graph-07956-66ccc97694-fwn8j\" (UID: \"9ebe67c6-85cc-45f1-a559-63329fdf7389\") " pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:21.471780 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.471743 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:21.506603 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.506563 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:21.645832 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:21.645642 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j"] Jul 09 18:11:21.648681 ip-10-0-141-169 kubenswrapper[2560]: W0709 18:11:21.648650 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ebe67c6_85cc_45f1_a559_63329fdf7389.slice/crio-4341165bb6cb901ac55cb52532e6fb9ecb863b3f74b126a6498ff07e2643f9ca WatchSource:0}: Error finding container 4341165bb6cb901ac55cb52532e6fb9ecb863b3f74b126a6498ff07e2643f9ca: Status 404 returned error can't find the container with id 4341165bb6cb901ac55cb52532e6fb9ecb863b3f74b126a6498ff07e2643f9ca Jul 09 18:11:22.363583 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:22.363545 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" event={"ID":"9ebe67c6-85cc-45f1-a559-63329fdf7389","Type":"ContainerStarted","Data":"56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0"} Jul 09 18:11:22.363995 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:22.363605 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" event={"ID":"9ebe67c6-85cc-45f1-a559-63329fdf7389","Type":"ContainerStarted","Data":"4341165bb6cb901ac55cb52532e6fb9ecb863b3f74b126a6498ff07e2643f9ca"} Jul 09 18:11:22.363995 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:22.363726 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:22.379689 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:22.379630 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" podStartSLOduration=1.379613285 podStartE2EDuration="1.379613285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:11:22.378666708 +0000 UTC m=+1910.081167899" watchObservedRunningTime="2026-07-09 18:11:22.379613285 +0000 UTC m=+1910.082114521" Jul 09 18:11:23.345167 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:23.345118 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:28.344584 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:28.344542 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:28.375003 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:28.374968 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:31.270612 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:31.270573 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j"] Jul 09 18:11:31.271028 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:31.270770 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" containerID="cri-o://56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0" gracePeriod=30 Jul 09 18:11:31.409317 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:31.409278 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:31.493773 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:31.493730 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:31.576628 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:31.576539 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:31.785204 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:31.785168 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:31.943462 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:31.943364 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:33.344586 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:33.344540 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:33.345076 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:33.344668 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:11:33.373631 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:33.373586 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:38.344816 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:38.344773 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:38.373944 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:38.373905 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:43.344559 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:43.344515 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:43.373015 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:43.372971 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:43.373195 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:43.373089 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:11:47.950302 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:47.950258 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:11:48.344890 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:48.344786 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:48.373985 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:48.373939 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:51.258493 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.258458 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:11:51.346852 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.346802 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d51d90-3774-4d90-8738-3c66b2a1a90b-openshift-service-ca-bundle\") pod \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\" (UID: \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\") " Jul 09 18:11:51.347043 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.346904 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7d51d90-3774-4d90-8738-3c66b2a1a90b-proxy-tls\") pod \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\" (UID: \"f7d51d90-3774-4d90-8738-3c66b2a1a90b\") " Jul 09 18:11:51.347254 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.347233 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d51d90-3774-4d90-8738-3c66b2a1a90b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "f7d51d90-3774-4d90-8738-3c66b2a1a90b" (UID: "f7d51d90-3774-4d90-8738-3c66b2a1a90b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:11:51.349189 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.349164 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7d51d90-3774-4d90-8738-3c66b2a1a90b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f7d51d90-3774-4d90-8738-3c66b2a1a90b" (UID: "f7d51d90-3774-4d90-8738-3c66b2a1a90b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:11:51.448301 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.448272 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f7d51d90-3774-4d90-8738-3c66b2a1a90b-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:11:51.448301 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.448303 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f7d51d90-3774-4d90-8738-3c66b2a1a90b-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:11:51.486471 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.486407 2560 generic.go:356] "Generic (PLEG): container finished" podID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerID="a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e" exitCode=0 Jul 09 18:11:51.486641 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.486477 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" Jul 09 18:11:51.486641 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.486485 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" event={"ID":"f7d51d90-3774-4d90-8738-3c66b2a1a90b","Type":"ContainerDied","Data":"a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e"} Jul 09 18:11:51.486641 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.486545 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw" event={"ID":"f7d51d90-3774-4d90-8738-3c66b2a1a90b","Type":"ContainerDied","Data":"d65c15dfdf5f1e796b3c4c3f74dfded34085a6125d405bfea1b776a9b9ad7770"} Jul 09 18:11:51.486641 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.486571 2560 scope.go:117] "RemoveContainer" containerID="a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e" Jul 09 18:11:51.497011 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.496982 2560 scope.go:117] "RemoveContainer" containerID="a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e" Jul 09 18:11:51.497694 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:11:51.497666 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e\": container with ID starting with a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e not found: ID does not exist" containerID="a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e" Jul 09 18:11:51.497816 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.497706 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e"} err="failed to get container status \"a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e\": rpc error: code = NotFound desc = could not find container \"a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e\": container with ID starting with a74ed86a04540868ec8d83cff261d190467b1bf49385d88ee8dce8214280179e not found: ID does not exist" Jul 09 18:11:51.506607 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.506580 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw"] Jul 09 18:11:51.509712 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:51.509687 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-67922-65cfc7d557-46nnw"] Jul 09 18:11:52.813315 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:52.813280 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" path="/var/lib/kubelet/pods/f7d51d90-3774-4d90-8738-3c66b2a1a90b/volumes" Jul 09 18:11:53.373571 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:53.373527 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:11:58.374155 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:11:58.374116 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:12:01.427990 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.427964 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:12:01.529495 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.529383 2560 generic.go:356] "Generic (PLEG): container finished" podID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerID="56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0" exitCode=0 Jul 09 18:12:01.529495 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.529461 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" event={"ID":"9ebe67c6-85cc-45f1-a559-63329fdf7389","Type":"ContainerDied","Data":"56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0"} Jul 09 18:12:01.529684 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.529519 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" event={"ID":"9ebe67c6-85cc-45f1-a559-63329fdf7389","Type":"ContainerDied","Data":"4341165bb6cb901ac55cb52532e6fb9ecb863b3f74b126a6498ff07e2643f9ca"} Jul 09 18:12:01.529684 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.529467 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j" Jul 09 18:12:01.529684 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.529541 2560 scope.go:117] "RemoveContainer" containerID="56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0" Jul 09 18:12:01.531687 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.531669 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebe67c6-85cc-45f1-a559-63329fdf7389-openshift-service-ca-bundle\") pod \"9ebe67c6-85cc-45f1-a559-63329fdf7389\" (UID: \"9ebe67c6-85cc-45f1-a559-63329fdf7389\") " Jul 09 18:12:01.531782 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.531735 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ebe67c6-85cc-45f1-a559-63329fdf7389-proxy-tls\") pod \"9ebe67c6-85cc-45f1-a559-63329fdf7389\" (UID: \"9ebe67c6-85cc-45f1-a559-63329fdf7389\") " Jul 09 18:12:01.532083 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.532055 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ebe67c6-85cc-45f1-a559-63329fdf7389-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "9ebe67c6-85cc-45f1-a559-63329fdf7389" (UID: "9ebe67c6-85cc-45f1-a559-63329fdf7389"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:12:01.533849 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.533815 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ebe67c6-85cc-45f1-a559-63329fdf7389-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ebe67c6-85cc-45f1-a559-63329fdf7389" (UID: "9ebe67c6-85cc-45f1-a559-63329fdf7389"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:12:01.543459 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.543429 2560 scope.go:117] "RemoveContainer" containerID="56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0" Jul 09 18:12:01.543830 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:12:01.543811 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0\": container with ID starting with 56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0 not found: ID does not exist" containerID="56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0" Jul 09 18:12:01.543882 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.543841 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0"} err="failed to get container status \"56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0\": rpc error: code = NotFound desc = could not find container \"56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0\": container with ID starting with 56272d5ff66a3433d21a26be7f8a7a7d7767e7f77d95fbb19d6ce5f107221ce0 not found: ID does not exist" Jul 09 18:12:01.632775 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.632735 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ebe67c6-85cc-45f1-a559-63329fdf7389-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:12:01.632775 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.632767 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ebe67c6-85cc-45f1-a559-63329fdf7389-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:12:01.850179 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.850142 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j"] Jul 09 18:12:01.853546 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:01.853520 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-07956-66ccc97694-fwn8j"] Jul 09 18:12:02.811596 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:02.811558 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" path="/var/lib/kubelet/pods/9ebe67c6-85cc-45f1-a559-63329fdf7389/volumes" Jul 09 18:12:31.344889 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.344850 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl"] Jul 09 18:12:31.345272 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.345153 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" Jul 09 18:12:31.345272 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.345166 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" Jul 09 18:12:31.345272 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.345177 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" Jul 09 18:12:31.345272 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.345183 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" Jul 09 18:12:31.345272 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.345247 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ebe67c6-85cc-45f1-a559-63329fdf7389" containerName="splitter-graph-07956" Jul 09 18:12:31.345272 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.345256 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d51d90-3774-4d90-8738-3c66b2a1a90b" containerName="sequence-graph-67922" Jul 09 18:12:31.348047 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.348021 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:31.351115 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.351088 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-5d567-serving-cert\"" Jul 09 18:12:31.351230 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.351182 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mspnj\"" Jul 09 18:12:31.351486 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.351462 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-5d567-kube-rbac-proxy-sar-config\"" Jul 09 18:12:31.351593 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.351578 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 09 18:12:31.360011 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.359985 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl"] Jul 09 18:12:31.360169 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.360115 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:12:31.488829 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.488786 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls\") pod \"switch-graph-5d567-6799fc7b-xq5cl\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:31.489023 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.488907 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d548b75-56ac-400b-9dc5-d4e8f576990e-openshift-service-ca-bundle\") pod \"switch-graph-5d567-6799fc7b-xq5cl\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:31.489286 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.489263 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd"] Jul 09 18:12:31.492713 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.492690 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:31.494714 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.494694 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-55ba8-kube-rbac-proxy-sar-config\"" Jul 09 18:12:31.494844 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.494729 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-55ba8-serving-cert\"" Jul 09 18:12:31.506021 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.505898 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:12:31.507078 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.507052 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd"] Jul 09 18:12:31.589748 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.589713 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls\") pod \"switch-graph-5d567-6799fc7b-xq5cl\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:31.589748 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.589755 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4515bb6-c307-4d3b-b5cf-51835d301110-openshift-service-ca-bundle\") pod \"splitter-graph-55ba8-6dcb4fc9cc-gvgrd\" (UID: \"b4515bb6-c307-4d3b-b5cf-51835d301110\") " pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:31.590025 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.589788 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4515bb6-c307-4d3b-b5cf-51835d301110-proxy-tls\") pod \"splitter-graph-55ba8-6dcb4fc9cc-gvgrd\" (UID: \"b4515bb6-c307-4d3b-b5cf-51835d301110\") " pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:31.590025 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.589854 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d548b75-56ac-400b-9dc5-d4e8f576990e-openshift-service-ca-bundle\") pod \"switch-graph-5d567-6799fc7b-xq5cl\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:31.590025 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:12:31.589895 2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-5d567-serving-cert: secret "switch-graph-5d567-serving-cert" not found Jul 09 18:12:31.590025 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:12:31.589986 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls podName:4d548b75-56ac-400b-9dc5-d4e8f576990e nodeName:}" failed. No retries permitted until 2026-07-09 18:12:32.089961841 +0000 UTC m=+1979.792463014 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls") pod "switch-graph-5d567-6799fc7b-xq5cl" (UID: "4d548b75-56ac-400b-9dc5-d4e8f576990e") : secret "switch-graph-5d567-serving-cert" not found Jul 09 18:12:31.590505 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.590485 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d548b75-56ac-400b-9dc5-d4e8f576990e-openshift-service-ca-bundle\") pod \"switch-graph-5d567-6799fc7b-xq5cl\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:31.684868 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.684774 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:12:31.690553 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.690527 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4515bb6-c307-4d3b-b5cf-51835d301110-openshift-service-ca-bundle\") pod \"splitter-graph-55ba8-6dcb4fc9cc-gvgrd\" (UID: \"b4515bb6-c307-4d3b-b5cf-51835d301110\") " pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:31.690674 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.690572 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4515bb6-c307-4d3b-b5cf-51835d301110-proxy-tls\") pod \"splitter-graph-55ba8-6dcb4fc9cc-gvgrd\" (UID: \"b4515bb6-c307-4d3b-b5cf-51835d301110\") " pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:31.691192 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.691164 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4515bb6-c307-4d3b-b5cf-51835d301110-openshift-service-ca-bundle\") pod \"splitter-graph-55ba8-6dcb4fc9cc-gvgrd\" (UID: \"b4515bb6-c307-4d3b-b5cf-51835d301110\") " pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:31.693150 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.693118 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4515bb6-c307-4d3b-b5cf-51835d301110-proxy-tls\") pod \"splitter-graph-55ba8-6dcb4fc9cc-gvgrd\" (UID: \"b4515bb6-c307-4d3b-b5cf-51835d301110\") " pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:31.813314 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.813266 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:31.953802 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:31.953765 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd"] Jul 09 18:12:31.957654 ip-10-0-141-169 kubenswrapper[2560]: W0709 18:12:31.957619 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4515bb6_c307_4d3b_b5cf_51835d301110.slice/crio-f01d3f0feb9403c49d63b5512a2a579766d00277c93e8181b1e6d1742d491e60 WatchSource:0}: Error finding container f01d3f0feb9403c49d63b5512a2a579766d00277c93e8181b1e6d1742d491e60: Status 404 returned error can't find the container with id f01d3f0feb9403c49d63b5512a2a579766d00277c93e8181b1e6d1742d491e60 Jul 09 18:12:32.093612 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:32.093569 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls\") pod \"switch-graph-5d567-6799fc7b-xq5cl\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:32.093800 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:12:32.093736 2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-5d567-serving-cert: secret "switch-graph-5d567-serving-cert" not found Jul 09 18:12:32.093847 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:12:32.093809 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls podName:4d548b75-56ac-400b-9dc5-d4e8f576990e nodeName:}" failed. No retries permitted until 2026-07-09 18:12:33.093791148 +0000 UTC m=+1980.796292316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls") pod "switch-graph-5d567-6799fc7b-xq5cl" (UID: "4d548b75-56ac-400b-9dc5-d4e8f576990e") : secret "switch-graph-5d567-serving-cert" not found Jul 09 18:12:32.329565 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:32.329460 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:12:32.657065 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:32.656955 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" event={"ID":"b4515bb6-c307-4d3b-b5cf-51835d301110","Type":"ContainerStarted","Data":"66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a"} Jul 09 18:12:32.657065 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:32.657021 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" event={"ID":"b4515bb6-c307-4d3b-b5cf-51835d301110","Type":"ContainerStarted","Data":"f01d3f0feb9403c49d63b5512a2a579766d00277c93e8181b1e6d1742d491e60"} Jul 09 18:12:32.657723 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:32.657129 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:32.673163 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:32.673103 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" podStartSLOduration=1.6730839899999999 podStartE2EDuration="1.67308399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:32.671532848 +0000 UTC m=+1980.374034038" watchObservedRunningTime="2026-07-09 18:12:32.67308399 +0000 UTC m=+1980.375585227" Jul 09 18:12:33.103741 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:33.103701 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls\") pod \"switch-graph-5d567-6799fc7b-xq5cl\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:33.106293 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:33.106254 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls\") pod \"switch-graph-5d567-6799fc7b-xq5cl\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:33.161141 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:33.161102 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:33.300094 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:33.300060 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl"] Jul 09 18:12:33.303162 ip-10-0-141-169 kubenswrapper[2560]: W0709 18:12:33.303131 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d548b75_56ac_400b_9dc5_d4e8f576990e.slice/crio-cae967b30c8535c0ec561db2875f1be3b7fa2915409d7256cfcd3d2f70d3ef2a WatchSource:0}: Error finding container cae967b30c8535c0ec561db2875f1be3b7fa2915409d7256cfcd3d2f70d3ef2a: Status 404 returned error can't find the container with id cae967b30c8535c0ec561db2875f1be3b7fa2915409d7256cfcd3d2f70d3ef2a Jul 09 18:12:33.662875 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:33.662773 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" event={"ID":"4d548b75-56ac-400b-9dc5-d4e8f576990e","Type":"ContainerStarted","Data":"2c5a161ba5b3859a11fbdf9152d2f1312cb1c5625f9c92689c4fca501fd055ef"} Jul 09 18:12:33.662875 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:33.662829 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" event={"ID":"4d548b75-56ac-400b-9dc5-d4e8f576990e","Type":"ContainerStarted","Data":"cae967b30c8535c0ec561db2875f1be3b7fa2915409d7256cfcd3d2f70d3ef2a"} Jul 09 18:12:33.679591 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:33.679532 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" podStartSLOduration=2.679513687 podStartE2EDuration="2.679513687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:12:33.678629772 +0000 UTC m=+1981.381130973" watchObservedRunningTime="2026-07-09 18:12:33.679513687 +0000 UTC m=+1981.382014876" Jul 09 18:12:34.667146 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:34.667111 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:12:38.669033 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:38.668997 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:12:40.676556 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:12:40.676523 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:13:47.654095 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:13:47.654059 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:15:02.675847 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:15:02.675809 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:17:47.538061 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:17:47.538022 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:20:39.353285 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:39.353238 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:20:39.438601 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:39.438565 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:20:45.932662 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:45.932617 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd"] Jul 09 18:20:45.933155 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:45.932900 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" containerID="cri-o://66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a" gracePeriod=30 Jul 09 18:20:45.959924 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:45.959868 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:20:46.101334 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:46.101249 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:20:46.174201 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:46.174161 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:20:48.666785 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:48.666745 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:20:53.667689 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:53.667649 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:20:58.667390 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:58.667334 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:20:58.667873 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:20:58.667528 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:21:03.667331 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:03.667292 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:21:08.667412 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:08.667368 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:21:13.667478 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:13.667409 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:21:16.091340 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.091308 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:21:16.165936 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.165899 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4515bb6-c307-4d3b-b5cf-51835d301110-proxy-tls\") pod \"b4515bb6-c307-4d3b-b5cf-51835d301110\" (UID: \"b4515bb6-c307-4d3b-b5cf-51835d301110\") " Jul 09 18:21:16.166108 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.165955 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4515bb6-c307-4d3b-b5cf-51835d301110-openshift-service-ca-bundle\") pod \"b4515bb6-c307-4d3b-b5cf-51835d301110\" (UID: \"b4515bb6-c307-4d3b-b5cf-51835d301110\") " Jul 09 18:21:16.166330 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.166305 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4515bb6-c307-4d3b-b5cf-51835d301110-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "b4515bb6-c307-4d3b-b5cf-51835d301110" (UID: "b4515bb6-c307-4d3b-b5cf-51835d301110"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:21:16.167970 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.167947 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4515bb6-c307-4d3b-b5cf-51835d301110-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4515bb6-c307-4d3b-b5cf-51835d301110" (UID: "b4515bb6-c307-4d3b-b5cf-51835d301110"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:21:16.267191 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.267152 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4515bb6-c307-4d3b-b5cf-51835d301110-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:21:16.267191 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.267186 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4515bb6-c307-4d3b-b5cf-51835d301110-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:21:16.768736 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.768698 2560 generic.go:356] "Generic (PLEG): container finished" podID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerID="66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a" exitCode=0 Jul 09 18:21:16.768948 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.768753 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" Jul 09 18:21:16.768948 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.768790 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" event={"ID":"b4515bb6-c307-4d3b-b5cf-51835d301110","Type":"ContainerDied","Data":"66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a"} Jul 09 18:21:16.768948 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.768847 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd" event={"ID":"b4515bb6-c307-4d3b-b5cf-51835d301110","Type":"ContainerDied","Data":"f01d3f0feb9403c49d63b5512a2a579766d00277c93e8181b1e6d1742d491e60"} Jul 09 18:21:16.768948 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.768870 2560 scope.go:117] "RemoveContainer" containerID="66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a" Jul 09 18:21:16.777544 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.777526 2560 scope.go:117] "RemoveContainer" containerID="66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a" Jul 09 18:21:16.777857 ip-10-0-141-169 kubenswrapper[2560]: E0709 18:21:16.777838 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a\": container with ID starting with 66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a not found: ID does not exist" containerID="66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a" Jul 09 18:21:16.777925 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.777869 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a"} err="failed to get container status \"66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a\": rpc error: code = NotFound desc = could not find container \"66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a\": container with ID starting with 66506cdddf07bf821e71ac8f75fd9af72db8f6e33abd814bc502c97c244e684a not found: ID does not exist" Jul 09 18:21:16.787886 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.787853 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd"] Jul 09 18:21:16.790340 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.790301 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-55ba8-6dcb4fc9cc-gvgrd"] Jul 09 18:21:16.811945 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:21:16.811914 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" path="/var/lib/kubelet/pods/b4515bb6-c307-4d3b-b5cf-51835d301110/volumes" Jul 09 18:23:27.217404 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:23:27.217322 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:23:53.338142 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:23:53.338102 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:50.452879 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:50.452839 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl"] Jul 09 18:28:50.455469 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:50.453101 2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" containerID="cri-o://2c5a161ba5b3859a11fbdf9152d2f1312cb1c5625f9c92689c4fca501fd055ef" gracePeriod=30 Jul 09 18:28:50.522280 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:50.522244 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:50.676074 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:50.676026 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:28:51.827666 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:51.827629 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:52.129524 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:52.129399 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:52.185791 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:52.185756 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:28:55.675087 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:55.674998 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:28:58.910034 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:28:58.909995 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:00.675588 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:00.675537 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:29:00.676085 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:00.675652 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:29:05.675505 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:05.675459 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:29:07.372400 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:07.372360 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:08.201108 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:08.201078 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:09.025774 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:09.025745 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:09.802472 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:09.802414 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:10.593730 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:10.593683 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:10.674912 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:10.674871 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:29:11.397051 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:11.397017 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:12.171077 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:12.171036 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:12.955329 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:12.955292 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:13.752562 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:13.752532 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:14.533338 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:14.533302 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:15.337926 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:15.337892 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:15.675478 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:15.675339 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 18:29:16.114249 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:16.114219 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:16.916359 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:16.916326 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:17.736816 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:17.736788 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-5d567-6799fc7b-xq5cl_4d548b75-56ac-400b-9dc5-d4e8f576990e/switch-graph-5d567/0.log" Jul 09 18:29:20.563868 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.563831 2560 generic.go:356] "Generic (PLEG): container finished" podID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerID="2c5a161ba5b3859a11fbdf9152d2f1312cb1c5625f9c92689c4fca501fd055ef" exitCode=0 Jul 09 18:29:20.564310 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.563876 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" event={"ID":"4d548b75-56ac-400b-9dc5-d4e8f576990e","Type":"ContainerDied","Data":"2c5a161ba5b3859a11fbdf9152d2f1312cb1c5625f9c92689c4fca501fd055ef"} Jul 09 18:29:20.595891 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.595861 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:29:20.702293 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.702255 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls\") pod \"4d548b75-56ac-400b-9dc5-d4e8f576990e\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " Jul 09 18:29:20.702533 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.702325 2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d548b75-56ac-400b-9dc5-d4e8f576990e-openshift-service-ca-bundle\") pod \"4d548b75-56ac-400b-9dc5-d4e8f576990e\" (UID: \"4d548b75-56ac-400b-9dc5-d4e8f576990e\") " Jul 09 18:29:20.702755 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.702723 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d548b75-56ac-400b-9dc5-d4e8f576990e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "4d548b75-56ac-400b-9dc5-d4e8f576990e" (UID: "4d548b75-56ac-400b-9dc5-d4e8f576990e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 18:29:20.704655 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.704627 2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4d548b75-56ac-400b-9dc5-d4e8f576990e" (UID: "4d548b75-56ac-400b-9dc5-d4e8f576990e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 18:29:20.803738 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.803682 2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4d548b75-56ac-400b-9dc5-d4e8f576990e-proxy-tls\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:29:20.803738 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:20.803731 2560 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d548b75-56ac-400b-9dc5-d4e8f576990e-openshift-service-ca-bundle\") on node \"ip-10-0-141-169.ec2.internal\" DevicePath \"\"" Jul 09 18:29:21.568480 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:21.568451 2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" Jul 09 18:29:21.568904 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:21.568444 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl" event={"ID":"4d548b75-56ac-400b-9dc5-d4e8f576990e","Type":"ContainerDied","Data":"cae967b30c8535c0ec561db2875f1be3b7fa2915409d7256cfcd3d2f70d3ef2a"} Jul 09 18:29:21.568904 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:21.568608 2560 scope.go:117] "RemoveContainer" containerID="2c5a161ba5b3859a11fbdf9152d2f1312cb1c5625f9c92689c4fca501fd055ef" Jul 09 18:29:21.584852 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:21.584819 2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl"] Jul 09 18:29:21.588978 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:21.588945 2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-5d567-6799fc7b-xq5cl"] Jul 09 18:29:22.811301 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:22.811264 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" path="/var/lib/kubelet/pods/4d548b75-56ac-400b-9dc5-d4e8f576990e/volumes" Jul 09 18:29:23.226496 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:23.226470 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:23.393249 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:23.393198 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:23.491341 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:23.491249 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:25.830562 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:25.830525 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-7wrgh_c5ff51e9-bbc2-419e-84f4-44b56c5ae8d7/global-pull-secret-syncer/0.log" Jul 09 18:29:26.004139 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:26.004101 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" Jul 09 18:29:26.224995 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:26.224958 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-169.ec2.internal_2a18d5082e7af3d615bed4125b1e112d/haproxy/0.log" Jul 09 18:29:29.238616 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:29.238577 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:29.338288 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:29.338256 2560 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-n2vlq_5287d7be-202c-41e9-99a8-5248fce6c989/konnectivity-agent/0.log" limit=1024 Jul 09 18:29:30.339758 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:30.339727 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dmdv5_08596c60-5055-4866-a866-18b83867ba3c/node-exporter/0.log" Jul 09 18:29:30.364710 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:30.364683 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dmdv5_08596c60-5055-4866-a866-18b83867ba3c/kube-rbac-proxy/0.log" Jul 09 18:29:30.392247 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:30.392220 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-dmdv5_08596c60-5055-4866-a866-18b83867ba3c/init-textfile/0.log" Jul 09 18:29:33.538022 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:33.537973 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57485bcffb-lk47g_7576a070-9722-44b8-8331-63f928ed6b68/console/0.log" Jul 09 18:29:34.208602 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.208556 2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj"] Jul 09 18:29:34.208941 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.208923 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" Jul 09 18:29:34.209002 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.208945 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" Jul 09 18:29:34.209002 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.208976 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" Jul 09 18:29:34.209002 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.208985 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" Jul 09 18:29:34.209108 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.209058 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4515bb6-c307-4d3b-b5cf-51835d301110" containerName="splitter-graph-55ba8" Jul 09 18:29:34.209108 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.209068 2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d548b75-56ac-400b-9dc5-d4e8f576990e" containerName="switch-graph-5d567" Jul 09 18:29:34.213197 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.213170 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.215472 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.215446 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q7djf\"/\"kube-root-ca.crt\"" Jul 09 18:29:34.215948 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.215927 2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-q7djf\"/\"default-dockercfg-n8q72\"" Jul 09 18:29:34.215948 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.215945 2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-q7djf\"/\"openshift-service-ca.crt\"" Jul 09 18:29:34.223555 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.223528 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj"] Jul 09 18:29:34.311131 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.311092 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-proc\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.311131 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.311136 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-sys\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.311360 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.311153 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwzw\" (UniqueName: \"kubernetes.io/projected/f1a555be-38a8-4b87-ac4c-6484daf6862e-kube-api-access-tbwzw\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.311360 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.311211 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-podres\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.311360 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.311260 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-lib-modules\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.411917 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.411871 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-proc\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.411917 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.411916 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-sys\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.412139 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.411946 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwzw\" (UniqueName: \"kubernetes.io/projected/f1a555be-38a8-4b87-ac4c-6484daf6862e-kube-api-access-tbwzw\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.412139 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.411982 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-podres\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.412139 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.411997 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-proc\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.412139 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.412014 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-sys\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.412139 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.412042 2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-lib-modules\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.412139 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.412102 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-podres\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.412330 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.412174 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f1a555be-38a8-4b87-ac4c-6484daf6862e-lib-modules\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.421097 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.421059 2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwzw\" (UniqueName: \"kubernetes.io/projected/f1a555be-38a8-4b87-ac4c-6484daf6862e-kube-api-access-tbwzw\") pod \"perf-node-gather-daemonset-grplj\" (UID: \"f1a555be-38a8-4b87-ac4c-6484daf6862e\") " pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.523961 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.523851 2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:34.660652 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:34.660615 2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj"] Jul 09 18:29:34.664244 ip-10-0-141-169 kubenswrapper[2560]: W0709 18:29:34.664210 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf1a555be_38a8_4b87_ac4c_6484daf6862e.slice/crio-961cd1cb689d326b4f9dd32db1cf9ae3b1ab50d2b267bef795f6e2466899219a WatchSource:0}: Error finding container 961cd1cb689d326b4f9dd32db1cf9ae3b1ab50d2b267bef795f6e2466899219a: Status 404 returned error can't find the container with id 961cd1cb689d326b4f9dd32db1cf9ae3b1ab50d2b267bef795f6e2466899219a Jul 09 18:29:35.101063 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:35.100978 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6dtwp_40f09dde-c76e-410b-98cd-cdb1de2858d4/dns/0.log" Jul 09 18:29:35.126107 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:35.126082 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6dtwp_40f09dde-c76e-410b-98cd-cdb1de2858d4/kube-rbac-proxy/0.log" Jul 09 18:29:35.280199 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:35.280164 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kzm26_1c43ac19-a9e4-47c7-a4ca-a5d758033df8/dns-node-resolver/0.log" Jul 09 18:29:35.624089 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:35.624050 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" event={"ID":"f1a555be-38a8-4b87-ac4c-6484daf6862e","Type":"ContainerStarted","Data":"77f262381494529e68cc4a46750c6bae2a1365961582e01f3e1c08ce2c035099"} Jul 09 18:29:35.624295 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:35.624111 2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" event={"ID":"f1a555be-38a8-4b87-ac4c-6484daf6862e","Type":"ContainerStarted","Data":"961cd1cb689d326b4f9dd32db1cf9ae3b1ab50d2b267bef795f6e2466899219a"} Jul 09 18:29:35.624295 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:35.624203 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:35.639978 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:35.639920 2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" podStartSLOduration=1.639902911 podStartE2EDuration="1.639902911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:29:35.638723785 +0000 UTC m=+3003.341224976" watchObservedRunningTime="2026-07-09 18:29:35.639902911 +0000 UTC m=+3003.342404078" Jul 09 18:29:35.841351 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:35.841318 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4b8tl_a0114f7e-8bb8-40ce-a472-15872a204f8a/node-ca/0.log" Jul 09 18:29:37.134980 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:37.134947 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mcntj_aca88050-a7a2-4388-b3bd-20c66b173963/serve-healthcheck-canary/0.log" Jul 09 18:29:37.667717 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:37.667675 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xj9gk_ab7fc605-af53-4640-bda0-d873c2a53eeb/kube-rbac-proxy/0.log" Jul 09 18:29:37.687710 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:37.687685 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xj9gk_ab7fc605-af53-4640-bda0-d873c2a53eeb/exporter/0.log" Jul 09 18:29:37.709769 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:37.709741 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xj9gk_ab7fc605-af53-4640-bda0-d873c2a53eeb/extractor/0.log" Jul 09 18:29:39.770937 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:39.770901 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6b8548bc6d-9ff4t_20a09a93-93d8-4a3c-826b-9dbe3b0f1792/manager/0.log" Jul 09 18:29:39.792400 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:39.792350 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-654c557856-x5h68_b4fb0bcc-5a67-4a91-9183-f53c40cd1cd2/manager/0.log" Jul 09 18:29:41.637337 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:41.637312 2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-q7djf/perf-node-gather-daemonset-grplj" Jul 09 18:29:45.961707 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:45.961680 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzjj6_9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48/kube-multus-additional-cni-plugins/0.log" Jul 09 18:29:45.985994 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:45.985967 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzjj6_9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48/egress-router-binary-copy/0.log" Jul 09 18:29:46.009869 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:46.009847 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzjj6_9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48/cni-plugins/0.log" Jul 09 18:29:46.032930 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:46.032909 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzjj6_9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48/bond-cni-plugin/0.log" Jul 09 18:29:46.057658 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:46.057636 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzjj6_9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48/routeoverride-cni/0.log" Jul 09 18:29:46.081360 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:46.081326 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzjj6_9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48/whereabouts-cni-bincopy/0.log" Jul 09 18:29:46.106459 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:46.106433 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rzjj6_9e3a4288-a9f1-444c-a3cd-e0bfd9f86e48/whereabouts-cni/0.log" Jul 09 18:29:46.170590 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:46.170557 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l62j2_0fbeb1a7-728c-4b64-aaba-5e1e3a1c1e98/kube-multus/0.log" Jul 09 18:29:46.190710 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:46.190683 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7vlf2_86dbd8b7-a392-48da-901a-a23b1fabec38/network-metrics-daemon/0.log" Jul 09 18:29:46.211554 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:46.211528 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7vlf2_86dbd8b7-a392-48da-901a-a23b1fabec38/kube-rbac-proxy/0.log" Jul 09 18:29:47.075343 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:47.075308 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xxd4_896c18d4-5355-4f6a-80c4-017b47ab37ad/ovn-controller/0.log" Jul 09 18:29:47.098973 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:47.098950 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xxd4_896c18d4-5355-4f6a-80c4-017b47ab37ad/ovn-acl-logging/0.log" Jul 09 18:29:47.125404 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:47.125382 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xxd4_896c18d4-5355-4f6a-80c4-017b47ab37ad/kube-rbac-proxy-node/0.log" Jul 09 18:29:47.152223 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:47.152197 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xxd4_896c18d4-5355-4f6a-80c4-017b47ab37ad/kube-rbac-proxy-ovn-metrics/0.log" Jul 09 18:29:47.174637 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:47.174608 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xxd4_896c18d4-5355-4f6a-80c4-017b47ab37ad/northd/0.log" Jul 09 18:29:47.200012 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:47.199982 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xxd4_896c18d4-5355-4f6a-80c4-017b47ab37ad/nbdb/0.log" Jul 09 18:29:47.222082 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:47.222057 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xxd4_896c18d4-5355-4f6a-80c4-017b47ab37ad/sbdb/0.log" Jul 09 18:29:47.260104 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:47.260066 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8xxd4_896c18d4-5355-4f6a-80c4-017b47ab37ad/ovnkube-controller/0.log" Jul 09 18:29:48.736488 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:48.736459 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-869dcfcdcd-pbjck_53a6095e-30c2-416a-8bd3-136362053252/check-endpoints/0.log" Jul 09 18:29:48.816318 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:48.816284 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-rdfxs_3854aae9-5583-4345-a96c-9f3f2f90cbb3/network-check-target-container/0.log" Jul 09 18:29:49.722166 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:49.722140 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-sq4tk_a281f32e-f58f-45fd-936d-96e8de5c6121/iptables-alerter/0.log" Jul 09 18:29:50.356508 ip-10-0-141-169 kubenswrapper[2560]: I0709 18:29:50.356477 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-4lhfl_22257c40-5297-4b5a-b377-e7db21adfc99/tuned/0.log"