Sep 16 18:45:12.096236 ip-10-0-140-230 systemd[1]: Starting Kubernetes Kubelet... Sep 16 18:45:12.486984 ip-10-0-140-230 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:12.486984 ip-10-0-140-230 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 18:45:12.486984 ip-10-0-140-230 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:12.486984 ip-10-0-140-230 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 18:45:12.486984 ip-10-0-140-230 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:12.488739 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.488659 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 18:45:12.492179 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492165 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.492179 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492179 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492183 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492186 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492190 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492193 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492196 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492199 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492203 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492206 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492208 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492211 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492214 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492216 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492222 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492225 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492228 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492230 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492233 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492235 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.492239 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492238 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492241 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492244 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492247 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492250 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492253 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492255 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492258 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492261 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492264 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492266 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492269 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492271 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492274 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492276 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492279 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492281 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492283 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492293 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492296 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.492737 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492298 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492301 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492303 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492305 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492308 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492310 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492314 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492316 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492319 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492322 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492325 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492327 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492330 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492332 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492336 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492338 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492341 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492344 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492348 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.493275 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492352 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492355 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492359 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492363 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492365 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492368 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492371 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492374 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492377 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492379 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492383 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492386 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492389 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492392 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492395 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492398 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492400 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492403 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492406 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.494095 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492409 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492414 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492417 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492419 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492422 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492424 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492427 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492430 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492433 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492436 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492438 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492441 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492443 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492446 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492448 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492451 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492453 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492456 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492458 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492461 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.494611 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.492464 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493538 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493553 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493557 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493562 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493567 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493571 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493575 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493579 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493584 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493588 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493598 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493603 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493607 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493612 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493617 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493621 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493625 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493629 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493633 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.495119 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493639 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493644 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493648 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493652 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493661 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493667 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493672 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493677 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493681 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493685 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493689 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493694 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493698 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493702 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493707 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493711 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493715 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493724 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493728 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493733 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.495665 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493737 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493741 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493745 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493750 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493754 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493758 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493763 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493768 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493772 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493777 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493786 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493791 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493796 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493800 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493804 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493809 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493813 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493817 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493821 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493825 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.496176 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493830 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493835 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493844 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493849 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493853 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493858 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493862 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493867 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493871 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493875 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493883 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493890 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493894 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493899 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493908 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493913 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493918 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493924 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493932 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.496667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493937 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493943 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493948 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493953 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493958 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493984 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493989 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493993 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.493997 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494002 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494006 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494011 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494015 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494019 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494023 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494028 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494032 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494036 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494045 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494050 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.497157 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494053 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494677 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494686 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494690 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494694 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494697 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494700 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494703 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494706 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494709 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494711 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494714 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494717 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494720 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494722 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494725 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494728 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494731 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494735 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494739 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.498387 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494742 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494744 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494747 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494750 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494753 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494755 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494758 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494762 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494765 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494768 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494771 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494773 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494776 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494779 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494782 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494785 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494788 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494790 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494793 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494795 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.498864 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494798 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494800 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494803 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494805 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494808 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494810 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494813 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494815 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494818 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494820 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494823 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494825 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494828 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494830 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494834 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494838 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494841 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494843 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494846 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494848 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.499431 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494851 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494854 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494856 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494859 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494861 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494864 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494867 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494870 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494873 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494875 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494878 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494880 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494883 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494885 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494888 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494890 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494892 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494895 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494897 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.499922 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494900 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494902 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494904 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494907 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494910 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494912 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494915 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494917 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494920 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494922 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494925 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494928 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494930 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494933 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494935 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494941 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494943 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494946 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494950 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.500410 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494953 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.494955 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495044 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495081 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495086 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495091 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495095 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495098 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495102 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495107 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495110 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495113 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495116 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495119 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495123 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495126 2559 flags.go:64] FLAG: --cgroup-root="" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495129 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495132 2559 flags.go:64] FLAG: --client-ca-file="" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495136 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495139 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495143 2559 flags.go:64] FLAG: --cluster-domain="" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495146 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495149 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495152 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 18:45:12.500882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495156 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495160 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495163 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495166 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495169 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495172 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495175 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495179 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495182 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495185 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495189 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495192 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495195 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495197 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495200 2559 flags.go:64] FLAG: --enable-server="true" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495203 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495207 2559 flags.go:64] FLAG: --event-burst="100" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495210 2559 flags.go:64] FLAG: --event-qps="50" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495213 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495216 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495218 2559 flags.go:64] FLAG: --eviction-hard="" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495222 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495225 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495228 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495230 2559 flags.go:64] FLAG: --eviction-soft="" Sep 16 18:45:12.501472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495233 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495236 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495239 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495242 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495245 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495248 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495251 2559 flags.go:64] FLAG: --feature-gates="" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495255 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495258 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495261 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495264 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495267 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495270 2559 flags.go:64] FLAG: --help="false" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495273 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495276 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495279 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495282 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495286 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495289 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495292 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495295 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495298 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495301 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 18:45:12.502115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495304 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495307 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495310 2559 flags.go:64] FLAG: --kube-reserved="" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495312 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495315 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495318 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495322 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495325 2559 flags.go:64] FLAG: --lock-file="" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495327 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495330 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495333 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495339 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495342 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495345 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495348 2559 flags.go:64] FLAG: --logging-format="text" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495351 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495354 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495357 2559 flags.go:64] FLAG: --manifest-url="" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495360 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495364 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495367 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495371 2559 flags.go:64] FLAG: --max-pods="110" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495374 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495377 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495380 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 18:45:12.502733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495383 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495386 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495388 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495392 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495398 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495401 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495404 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495407 2559 flags.go:64] FLAG: --pod-cidr="" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495410 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495416 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495419 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495422 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495425 2559 flags.go:64] FLAG: --port="10250" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495427 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495430 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e186faeb9c689e94" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495433 2559 flags.go:64] FLAG: --qos-reserved="" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495436 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495439 2559 flags.go:64] FLAG: --register-node="true" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495442 2559 flags.go:64] FLAG: --register-with-taints="" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495445 2559 flags.go:64] FLAG: --registry-burst="10" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495449 2559 flags.go:64] FLAG: --registry-qps="5" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495452 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495455 2559 flags.go:64] FLAG: --reserved-memory="" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495458 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495461 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 18:45:12.503364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495464 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495467 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495469 2559 flags.go:64] FLAG: --runonce="false" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495472 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495475 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495478 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495481 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495483 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495486 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495489 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495492 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495495 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495498 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495501 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495504 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495507 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495510 2559 flags.go:64] FLAG: --system-cgroups="" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495512 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495518 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495520 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495523 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495527 2559 flags.go:64] FLAG: --tls-min-version="" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495530 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495533 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495536 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 18:45:12.504000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495539 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495541 2559 flags.go:64] FLAG: --v="2" Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495550 2559 flags.go:64] FLAG: --version="false" Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495554 2559 flags.go:64] FLAG: --vmodule="" Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495560 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.495563 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495677 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495682 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495685 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495689 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495692 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495695 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495699 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495702 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495704 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495707 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495710 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495712 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495715 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495718 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495720 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.504615 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495723 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495726 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495728 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495731 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495735 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495739 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495741 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495744 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495747 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495750 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495752 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495755 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495758 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495760 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495763 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495767 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495769 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495772 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495775 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495777 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.505144 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495780 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495783 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495786 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495790 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495792 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495795 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495798 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495800 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495803 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495805 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495808 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495810 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495813 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495816 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495818 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495821 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495823 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495826 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495829 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495831 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.505667 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495834 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495836 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495839 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495842 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495844 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495847 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495849 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495854 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495857 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495860 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495862 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495865 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495868 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495870 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495874 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495882 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495885 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495888 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495892 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.506170 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495896 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495900 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495903 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495906 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495909 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495911 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495914 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495917 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495919 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495922 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495925 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495927 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495930 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495932 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495935 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495938 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495940 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495943 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495945 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495948 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.506638 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495952 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495954 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495974 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495979 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.495982 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.496707 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.503951 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.503980 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.504860 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.504871 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504927 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504933 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504936 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504940 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504942 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.507142 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504945 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504948 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504951 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504955 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504957 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504975 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504979 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504982 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504985 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504988 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504991 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504993 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.504997 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505001 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505004 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505007 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505010 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505013 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505016 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.507520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505019 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505022 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505025 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505027 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505030 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505033 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505036 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505039 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505042 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505045 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505048 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505050 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505053 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505055 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505058 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505061 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505064 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505066 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505069 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.507989 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505071 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505074 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505077 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505079 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505082 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505084 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505087 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505089 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505092 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505094 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505097 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505100 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505102 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505105 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505108 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505111 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505113 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505116 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505120 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.508460 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505123 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505126 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505129 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505131 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505134 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505136 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505139 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505143 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505147 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505150 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505154 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505157 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505160 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505163 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505166 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505169 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505172 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505175 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505178 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.508942 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505180 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505183 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505186 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505188 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505191 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505194 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505196 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505199 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505201 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505205 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505208 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505211 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505214 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505217 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505219 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505222 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505224 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.509520 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505227 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.505232 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505339 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505343 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505346 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505349 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505352 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505355 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505358 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505360 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505363 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505365 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505368 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505371 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505374 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:12.509937 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505376 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505379 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505381 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505384 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505386 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505389 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505391 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505394 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505396 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505399 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505402 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505404 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505408 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505410 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505413 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505417 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505421 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505424 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505426 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:12.510334 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505429 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505432 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505435 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505437 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505440 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505442 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505445 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505447 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505450 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505452 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505455 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505457 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505460 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505462 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505465 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505467 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505470 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505472 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505475 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505477 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:12.510799 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505480 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505482 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505484 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505487 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505490 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505493 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505496 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505498 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505501 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505504 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505506 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505509 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505511 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505514 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505517 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505519 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505522 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505525 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505527 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505530 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:12.511289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505532 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505535 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505538 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505540 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505542 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505545 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505547 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505550 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505553 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505556 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505558 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505561 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505563 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505566 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505568 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505571 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505575 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505578 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505581 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:12.511768 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505583 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505587 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505591 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505594 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505597 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505599 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505602 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:12.505605 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.505610 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.505704 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.507998 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.508989 2559 server.go:1019] "Starting client certificate rotation" Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.509075 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:45:12.512234 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.509862 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:45:12.512555 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.511834 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:45:12.516404 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.516385 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:45:12.526700 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.526682 2559 log.go:25] "Validated CRI v1 runtime API" Sep 16 18:45:12.531345 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.531326 2559 log.go:25] "Validated CRI v1 image API" Sep 16 18:45:12.532104 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.532087 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 18:45:12.535483 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.535465 2559 fs.go:135] Filesystem UUIDs: map[3d862d99-04f7-49c5-b579-88cf20f1ad26:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c90d3d00-dae8-4de7-b709-67dfac375541:/dev/nvme0n1p3] Sep 16 18:45:12.535531 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.535483 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 18:45:12.541348 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.541245 2559 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:12.539884031 +0000 UTC m=+0.339005881 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099912 MemoryCapacity:33164468224 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d7c91a7ea07737f1dbbb4727fddbf SystemUUID:ec2d7c91-a7ea-0773-7f1d-bbb4727fddbf BootID:26ee438b-8c9b-43f8-b0e9-888dc7e8c7af Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582234112 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:14:06:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:14:06:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:1e:d7:0a:4d:f6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164468224 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 18:45:12.541348 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.541343 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 18:45:12.541458 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.541418 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 18:45:12.543039 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.543022 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:45:12.544679 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.544656 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 18:45:12.544806 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.544683 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-230.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 18:45:12.545556 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.545544 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 18:45:12.545556 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.545557 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 18:45:12.545628 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.545565 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:45:12.546413 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.546403 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:45:12.547568 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.547559 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 18:45:12.552030 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552019 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:45:12.552152 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552142 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 18:45:12.552184 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552172 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 16 18:45:12.552211 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552184 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 18:45:12.552211 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552201 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 18:45:12.552265 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552214 2559 kubelet.go:398] "Adding apiserver pod source" Sep 16 18:45:12.552265 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552225 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 18:45:12.552584 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552573 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:45:12.552623 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.552592 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:45:12.555029 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.555011 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 18:45:12.557284 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.557270 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 18:45:12.557327 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.557302 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 18:45:12.560088 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560076 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 18:45:12.560127 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560095 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 18:45:12.560127 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560102 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 18:45:12.560127 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560107 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 18:45:12.560127 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560116 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 18:45:12.560127 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560122 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 18:45:12.560255 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560136 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 18:45:12.560255 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560142 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 18:45:12.560255 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560149 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 18:45:12.560255 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560154 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 18:45:12.560255 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560165 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 18:45:12.560255 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.560171 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 18:45:12.561134 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.561125 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 18:45:12.561171 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.561136 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 18:45:12.562989 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.562958 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 18:45:12.563051 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.562981 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 18:45:12.563118 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.563102 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-230.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 18:45:12.565025 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.565012 2559 server.go:1268] "Started kubelet" Sep 16 18:45:12.565119 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.565099 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 18:45:12.565203 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.565164 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 18:45:12.565294 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.565216 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 18:45:12.566386 ip-10-0-140-230 systemd[1]: Started Kubernetes Kubelet. Sep 16 18:45:12.567179 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.567165 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 18:45:12.569206 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.569192 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 16 18:45:12.572333 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.572258 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 18:45:12.576152 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.576135 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 18:45:12.576816 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.576800 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 18:45:12.576927 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.576908 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:12.577517 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577503 2559 factory.go:55] Registering systemd factory Sep 16 18:45:12.577517 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577518 2559 factory.go:223] Registration of the systemd container factory successfully Sep 16 18:45:12.577642 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577570 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 18:45:12.577642 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577572 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 18:45:12.577642 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577601 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 18:45:12.577757 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577685 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 16 18:45:12.577757 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577692 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 16 18:45:12.577757 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577741 2559 factory.go:153] Registering CRI-O factory Sep 16 18:45:12.577757 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577750 2559 factory.go:223] Registration of the crio container factory successfully Sep 16 18:45:12.577876 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577785 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 18:45:12.577876 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577802 2559 factory.go:103] Registering Raw factory Sep 16 18:45:12.577876 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.577812 2559 manager.go:1196] Started watching for new ooms in manager Sep 16 18:45:12.578204 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.578186 2559 manager.go:349] Starting recovery of all containers Sep 16 18:45:12.579459 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.579430 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 18:45:12.580864 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.579538 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-230.ec2.internal.18d5e1735e0c3bd7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-230.ec2.internal,UID:ip-10-0-140-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-230.ec2.internal,},FirstTimestamp:2026-09-16 18:45:12.564988887 +0000 UTC m=+0.364110717,LastTimestamp:2026-09-16 18:45:12.564988887 +0000 UTC m=+0.364110717,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-230.ec2.internal,}" Sep 16 18:45:12.585700 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.585590 2559 manager.go:354] Recovery completed Sep 16 18:45:12.589481 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.589462 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-230.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 18:45:12.591449 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.591435 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.593006 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.592989 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.593086 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.593018 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.593086 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.593028 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.593531 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.593517 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.593599 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.593518 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 18:45:12.593599 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.593544 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 18:45:12.593599 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.593558 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:45:12.594150 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.594134 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.594227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.594159 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.594227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.594170 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.595616 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.595599 2559 policy_none.go:49] "None policy: Start" Sep 16 18:45:12.595616 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.595616 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 18:45:12.595705 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.595626 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 18:45:12.597492 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.597480 2559 policy_none.go:47] "Start" Sep 16 18:45:12.599353 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.599338 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xr8nq" Sep 16 18:45:12.605082 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.605013 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-230.ec2.internal.18d5e1735fb7b8ed default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-230.ec2.internal,UID:ip-10-0-140-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-230.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-230.ec2.internal,},FirstTimestamp:2026-09-16 18:45:12.593004781 +0000 UTC m=+0.392126611,LastTimestamp:2026-09-16 18:45:12.593004781 +0000 UTC m=+0.392126611,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-230.ec2.internal,}" Sep 16 18:45:12.607850 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.607834 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xr8nq" Sep 16 18:45:12.615533 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.615471 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-230.ec2.internal.18d5e1735fb7fdfd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-230.ec2.internal,UID:ip-10-0-140-230.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-230.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-230.ec2.internal,},FirstTimestamp:2026-09-16 18:45:12.593022461 +0000 UTC m=+0.392144291,LastTimestamp:2026-09-16 18:45:12.593022461 +0000 UTC m=+0.392144291,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-230.ec2.internal,}" Sep 16 18:45:12.633505 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.633489 2559 manager.go:337] "Starting Device Plugin manager" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.633561 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.633574 2559 server.go:87] "Starting device plugin registration server" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.633830 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.633842 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.633943 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.634036 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.634046 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.634515 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 18:45:12.654581 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.634564 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:12.714316 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.714285 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 18:45:12.715567 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.715546 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 18:45:12.715670 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.715612 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 18:45:12.715670 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.715635 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 18:45:12.715778 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.715677 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 18:45:12.718487 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.718467 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:12.734700 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.734677 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.735516 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.735489 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.735608 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.735533 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.735608 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.735547 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.735608 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.735575 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.742566 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.742507 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.742566 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.742527 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-230.ec2.internal\": node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:12.762241 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.762225 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:12.815980 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.815950 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal"] Sep 16 18:45:12.816059 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.816041 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.817379 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.817366 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.817450 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.817391 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.817450 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.817401 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.819434 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.819422 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.819548 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.819534 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.819594 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.819561 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.820080 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.820066 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.820149 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.820095 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.820149 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.820067 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.820149 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.820110 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.820149 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.820128 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.820149 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.820141 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.822300 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.822286 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.822373 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.822308 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:12.823136 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.823120 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:12.823208 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.823150 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:12.823208 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.823164 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:12.843489 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.843470 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-230.ec2.internal\" not found" node="ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.847931 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.847916 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-230.ec2.internal\" not found" node="ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.863093 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.863080 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:12.880491 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.880478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb404389450f16abc78fd737f64d12f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"bb404389450f16abc78fd737f64d12f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.880552 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.880498 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb404389450f16abc78fd737f64d12f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"bb404389450f16abc78fd737f64d12f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.880552 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.880518 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d512deac78b026258432227d6ed92f58-config\") pod \"kube-apiserver-proxy-ip-10-0-140-230.ec2.internal\" (UID: \"d512deac78b026258432227d6ed92f58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.963667 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:12.963648 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:12.980814 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.980794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb404389450f16abc78fd737f64d12f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"bb404389450f16abc78fd737f64d12f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.980880 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.980822 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d512deac78b026258432227d6ed92f58-config\") pod \"kube-apiserver-proxy-ip-10-0-140-230.ec2.internal\" (UID: \"d512deac78b026258432227d6ed92f58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.980880 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.980840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb404389450f16abc78fd737f64d12f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"bb404389450f16abc78fd737f64d12f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.980959 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.980893 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb404389450f16abc78fd737f64d12f5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"bb404389450f16abc78fd737f64d12f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.980959 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.980924 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb404389450f16abc78fd737f64d12f5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal\" (UID: \"bb404389450f16abc78fd737f64d12f5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:12.980959 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:12.980904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d512deac78b026258432227d6ed92f58-config\") pod \"kube-apiserver-proxy-ip-10-0-140-230.ec2.internal\" (UID: \"d512deac78b026258432227d6ed92f58\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" Sep 16 18:45:13.064389 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:13.064300 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:13.145873 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.145838 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:13.150697 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.150684 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" Sep 16 18:45:13.164388 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:13.164363 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:13.265909 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:13.265862 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:13.369928 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:13.369842 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:13.470682 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:13.470645 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:13.509735 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.509699 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 18:45:13.510184 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.509864 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 18:45:13.571808 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:13.571775 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-230.ec2.internal\" not found" Sep 16 18:45:13.614173 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.614139 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:13 +0000 UTC" deadline="2028-06-05 11:29:18.002164 +0000 UTC" Sep 16 18:45:13.614173 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.614169 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15064h44m4.387998878s" Sep 16 18:45:13.616196 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.616180 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:13.659166 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.659138 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:13.677766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.677739 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" Sep 16 18:45:13.703705 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.703683 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:45:13.705511 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.705496 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" Sep 16 18:45:13.725436 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.725413 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:45:13.801274 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:13.801244 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb404389450f16abc78fd737f64d12f5.slice/crio-24013c77883de6786ef671b2c12f512ee7a9f928fe80e4783332dc0bcbba7d16 WatchSource:0}: Error finding container 24013c77883de6786ef671b2c12f512ee7a9f928fe80e4783332dc0bcbba7d16: Status 404 returned error can't find the container with id 24013c77883de6786ef671b2c12f512ee7a9f928fe80e4783332dc0bcbba7d16 Sep 16 18:45:13.801762 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:13.801740 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd512deac78b026258432227d6ed92f58.slice/crio-d3641e9177de26c67ad672df8f8550fa015552192097460d46a0c6610c729fc6 WatchSource:0}: Error finding container d3641e9177de26c67ad672df8f8550fa015552192097460d46a0c6610c729fc6: Status 404 returned error can't find the container with id d3641e9177de26c67ad672df8f8550fa015552192097460d46a0c6610c729fc6 Sep 16 18:45:13.806401 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.806383 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:45:13.828317 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:13.828293 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:14.320844 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.320815 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:14.554639 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.554609 2559 apiserver.go:52] "Watching apiserver" Sep 16 18:45:14.572905 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.572829 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 18:45:14.574840 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.574776 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-jr5jk","openshift-network-operator/iptables-alerter-dzfmw","openshift-ovn-kubernetes/ovnkube-node-v2qjb","kube-system/global-pull-secret-syncer-sz2r4","kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c","openshift-dns/node-resolver-57pgm","openshift-multus/multus-9gp8g","openshift-multus/network-metrics-daemon-x7nkj","kube-system/konnectivity-agent-f9pjr","openshift-cluster-node-tuning-operator/tuned-75g4k","openshift-image-registry/node-ca-m9mfd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal","openshift-multus/multus-additional-cni-plugins-xvmpj"] Sep 16 18:45:14.578354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.577372 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.578354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.578196 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 18:45:14.579835 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.579819 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.582278 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.582264 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.583716 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.582960 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.583716 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.583265 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mgw2g\"" Sep 16 18:45:14.584443 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.584158 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.586069 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.585188 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.586069 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.585468 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvp8q\"" Sep 16 18:45:14.586069 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.585940 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 18:45:14.586303 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.586148 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.587586 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.587565 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 18:45:14.587586 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.587578 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.587705 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.587638 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:14.589503 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.589402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:14.589503 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.589463 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:14.591561 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.591522 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.592229 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-system-cni-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592335 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592242 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46efde69-5c9c-4fee-956d-f2c2c5855225-cni-binary-copy\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592335 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-k8s-cni-cncf-io\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592335 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592318 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-multus-certs\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592511 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956fd3b6-79ea-4195-8ce8-35658bd3d7de-tmp-dir\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.592511 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592386 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-cnibin\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592511 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592425 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-cni-bin\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592511 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-kubelet\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592511 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-hostroot\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592521 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-cni-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592561 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-netns\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592584 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-conf-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592618 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqxg\" (UniqueName: \"kubernetes.io/projected/46efde69-5c9c-4fee-956d-f2c2c5855225-kube-api-access-rpqxg\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/956fd3b6-79ea-4195-8ce8-35658bd3d7de-hosts-file\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.592760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592685 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-socket-dir-parent\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-cni-multus\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.592760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592740 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-daemon-config\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.593190 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-etc-kubernetes\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.593190 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.592787 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-498m5\" (UniqueName: \"kubernetes.io/projected/956fd3b6-79ea-4195-8ce8-35658bd3d7de-kube-api-access-498m5\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.594397 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.594011 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:14.594397 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.594100 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:14.594697 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.594678 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.594920 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.594904 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 18:45:14.595212 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.595196 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 18:45:14.595530 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.595514 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 18:45:14.597140 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.596882 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.597140 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.597034 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.597279 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.597261 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.597482 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.597467 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mv9lz\"" Sep 16 18:45:14.598036 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.597693 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 18:45:14.599079 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.598702 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.599546 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.599507 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 18:45:14.600450 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.600430 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vfb2p\"" Sep 16 18:45:14.601424 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.601396 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:14.603103 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.602871 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 18:45:14.603196 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.603143 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 18:45:14.603537 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.603365 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l5pkz\"" Sep 16 18:45:14.603537 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.603432 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.604644 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.603767 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.604644 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.604243 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.604644 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.604344 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mgldg\"" Sep 16 18:45:14.604644 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.604519 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 18:45:14.604984 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.604802 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 18:45:14.604984 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.604818 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:45:14.604984 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.604827 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.605227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.605137 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rhzj9\"" Sep 16 18:45:14.605421 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.605405 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 18:45:14.605940 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.605918 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.609969 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.608521 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fnpc8\"" Sep 16 18:45:14.609969 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.608847 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.609969 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.608863 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.611378 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.610744 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 18:45:14.611378 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.610913 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-cgnpg\"" Sep 16 18:45:14.611378 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.610971 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 18:45:14.611378 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.611219 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:45:14.650629 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.649723 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5qk97" Sep 16 18:45:14.659896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.659504 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5qk97" Sep 16 18:45:14.679186 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.679158 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 18:45:14.693225 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-kubelet\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.693359 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693238 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-hostroot\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.693359 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693306 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysctl-d\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.693464 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693363 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-sys\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.693464 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-tuned\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.693464 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6mdj\" (UniqueName: \"kubernetes.io/projected/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-kube-api-access-q6mdj\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.693464 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693444 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-conf-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8nv\" (UniqueName: \"kubernetes.io/projected/99e68b22-58b0-4760-9209-9732571d499c-kube-api-access-qr8nv\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693504 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysconfig\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-var-lib-kubelet\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693552 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-log-socket\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-socket-dir-parent\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693601 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g78hx\" (UniqueName: \"kubernetes.io/projected/8ede9a85-51fc-4f63-905f-659eeb168b3f-kube-api-access-g78hx\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-etc-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.693665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-kubelet\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693673 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-ovn\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693701 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-k8s-cni-cncf-io\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693720 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-hostroot\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-host-slash\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693787 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693811 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-conf-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693900 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-k8s-cni-cncf-io\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693909 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-socket-dir-parent\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693812 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-run-netns\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.693969 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-cnibin\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-cni-bin\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-iptables-alerter-script\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694097 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-cnibin\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-systemd\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694132 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-cni-bin\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694142 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-host\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694191 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694218 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-env-overrides\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-cni-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqxg\" (UniqueName: \"kubernetes.io/projected/46efde69-5c9c-4fee-956d-f2c2c5855225-kube-api-access-rpqxg\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694303 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-kubelet-config\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694328 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysctl-conf\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694329 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-cni-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694354 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694506 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-etc-kubernetes\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694587 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-dbus\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694619 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-etc-kubernetes\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-registration-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694663 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-kubernetes\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cnibin\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.694896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46efde69-5c9c-4fee-956d-f2c2c5855225-cni-binary-copy\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694741 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-sys-fs\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-serviceca\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovn-node-metrics-cert\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694805 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgzx\" (UniqueName: \"kubernetes.io/projected/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-kube-api-access-8hgzx\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694825 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc-konnectivity-ca\") pod \"konnectivity-agent-f9pjr\" (UID: \"9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc\") " pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694866 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-netns\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694882 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/956fd3b6-79ea-4195-8ce8-35658bd3d7de-hosts-file\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb5bj\" (UniqueName: \"kubernetes.io/projected/3198a50c-14d2-4981-8d5d-7718876d54b1-kube-api-access-xb5bj\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694913 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-498m5\" (UniqueName: \"kubernetes.io/projected/956fd3b6-79ea-4195-8ce8-35658bd3d7de-kube-api-access-498m5\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694929 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fff2r\" (UniqueName: \"kubernetes.io/projected/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-kube-api-access-fff2r\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-modprobe-d\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694956 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-run\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.694981 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85wq\" (UniqueName: \"kubernetes.io/projected/4fa27ea5-e7d8-4be1-9c00-410f85826d80-kube-api-access-f85wq\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-var-lib-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.695834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695057 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovnkube-script-lib\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-multus-certs\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-slash\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-cni-netd\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-device-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695183 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovnkube-config\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695224 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-system-cni-dir\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695255 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-host\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695270 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-systemd\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695289 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-cni-multus\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695313 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-daemon-config\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695340 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ede9a85-51fc-4f63-905f-659eeb168b3f-tmp\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695376 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc-agent-certs\") pod \"konnectivity-agent-f9pjr\" (UID: \"9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc\") " pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:14.696690 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695392 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/46efde69-5c9c-4fee-956d-f2c2c5855225-cni-binary-copy\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-cni-bin\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695436 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-system-cni-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695452 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956fd3b6-79ea-4195-8ce8-35658bd3d7de-tmp-dir\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-socket-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-lib-modules\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695502 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-multus-certs\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695511 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-kubelet\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695545 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-systemd-units\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695594 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-node-log\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-run-netns\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.695707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-host-var-lib-cni-multus\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.696078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/46efde69-5c9c-4fee-956d-f2c2c5855225-system-cni-dir\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.696176 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/46efde69-5c9c-4fee-956d-f2c2c5855225-multus-daemon-config\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.696168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/956fd3b6-79ea-4195-8ce8-35658bd3d7de-hosts-file\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.697310 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.696312 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/956fd3b6-79ea-4195-8ce8-35658bd3d7de-tmp-dir\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.708594 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.708557 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 18:45:14.714269 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.714244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-498m5\" (UniqueName: \"kubernetes.io/projected/956fd3b6-79ea-4195-8ce8-35658bd3d7de-kube-api-access-498m5\") pod \"node-resolver-57pgm\" (UID: \"956fd3b6-79ea-4195-8ce8-35658bd3d7de\") " pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.714543 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.714518 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqxg\" (UniqueName: \"kubernetes.io/projected/46efde69-5c9c-4fee-956d-f2c2c5855225-kube-api-access-rpqxg\") pod \"multus-9gp8g\" (UID: \"46efde69-5c9c-4fee-956d-f2c2c5855225\") " pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.721128 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.721079 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" event={"ID":"bb404389450f16abc78fd737f64d12f5","Type":"ContainerStarted","Data":"24013c77883de6786ef671b2c12f512ee7a9f928fe80e4783332dc0bcbba7d16"} Sep 16 18:45:14.723334 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.723304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" event={"ID":"d512deac78b026258432227d6ed92f58","Type":"ContainerStarted","Data":"d3641e9177de26c67ad672df8f8550fa015552192097460d46a0c6610c729fc6"} Sep 16 18:45:14.795845 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.795811 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-device-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.796011 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.795858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.796011 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.795886 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovnkube-config\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796011 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.795913 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.796011 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.795938 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-system-cni-dir\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.796011 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.795962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-host\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.796011 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-systemd\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ede9a85-51fc-4f63-905f-659eeb168b3f-tmp\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.796227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796057 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.796227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc-agent-certs\") pod \"konnectivity-agent-f9pjr\" (UID: \"9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc\") " pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:14.796227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-cni-bin\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-socket-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.796227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796178 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-lib-modules\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.796227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-kubelet\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796228 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-host\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-systemd-units\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796265 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796291 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-node-log\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysctl-d\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796338 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-sys\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796360 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-tuned\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796383 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6mdj\" (UniqueName: \"kubernetes.io/projected/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-kube-api-access-q6mdj\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8nv\" (UniqueName: \"kubernetes.io/projected/99e68b22-58b0-4760-9209-9732571d499c-kube-api-access-qr8nv\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.796467 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796437 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysconfig\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796472 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-var-lib-kubelet\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-log-socket\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g78hx\" (UniqueName: \"kubernetes.io/projected/8ede9a85-51fc-4f63-905f-659eeb168b3f-kube-api-access-g78hx\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-etc-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-ovn\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796656 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-host-slash\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-run-netns\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.796770 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.797170 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-iptables-alerter-script\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.797170 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796800 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-systemd\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.797170 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796802 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.797170 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-host\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.797170 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.796897 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-systemd\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.797410 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovnkube-config\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.797525 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797420 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-ovn\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.797525 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797460 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-host\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.797525 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.797696 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-run-netns\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.797696 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797567 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-system-cni-dir\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.797696 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797606 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-host-slash\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.797696 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797612 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysconfig\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.797696 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797655 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-etc-selinux\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.797696 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797661 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:14.797696 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-env-overrides\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797705 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-kubelet-config\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysctl-conf\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797746 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.797751 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797763 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-dbus\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-registration-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-kubernetes\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.797817 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs podName:3198a50c-14d2-4981-8d5d-7718876d54b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.297795613 +0000 UTC m=+3.096917440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs") pod "network-metrics-daemon-x7nkj" (UID: "3198a50c-14d2-4981-8d5d-7718876d54b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-kubernetes\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cnibin\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-var-lib-kubelet\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797881 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-sys-fs\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-serviceca\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-log-socket\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797934 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovn-node-metrics-cert\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgzx\" (UniqueName: \"kubernetes.io/projected/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-kube-api-access-8hgzx\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.798051 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.797987 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797987 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.797984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.798046 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret podName:79ae6a44-0eb1-43cc-a9ff-63bc5342a979 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.29803158 +0000 UTC m=+3.097153403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret") pod "global-pull-secret-syncer-sz2r4" (UID: "79ae6a44-0eb1-43cc-a9ff-63bc5342a979") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-device-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc-konnectivity-ca\") pod \"konnectivity-agent-f9pjr\" (UID: \"9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc\") " pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798110 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-run-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb5bj\" (UniqueName: \"kubernetes.io/projected/3198a50c-14d2-4981-8d5d-7718876d54b1-kube-api-access-xb5bj\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798173 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fff2r\" (UniqueName: \"kubernetes.io/projected/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-kube-api-access-fff2r\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798199 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-modprobe-d\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-run\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cnibin\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85wq\" (UniqueName: \"kubernetes.io/projected/4fa27ea5-e7d8-4be1-9c00-410f85826d80-kube-api-access-f85wq\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-systemd-units\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-var-lib-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798303 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-etc-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.798972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovnkube-script-lib\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798350 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-slash\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-node-log\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798392 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-slash\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798405 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-cni-netd\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798446 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-sys-fs\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysctl-conf\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-sysctl-d\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798589 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-modprobe-d\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798618 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-cni-netd\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798639 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-kubelet-config\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-sys\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-run\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798763 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4fa27ea5-e7d8-4be1-9c00-410f85826d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-dbus\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798870 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-registration-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799206 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4fa27ea5-e7d8-4be1-9c00-410f85826d80-cni-binary-copy\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799323 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-serviceca\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.799989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-env-overrides\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799437 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99e68b22-58b0-4760-9209-9732571d499c-socket-dir\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799469 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-cni-bin\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799562 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-lib-modules\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799727 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc-konnectivity-ca\") pod \"konnectivity-agent-f9pjr\" (UID: \"9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc\") " pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.798320 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-var-lib-openvswitch\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-kubelet\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovnkube-script-lib\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.799923 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-systemd\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.800011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-iptables-alerter-script\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.800868 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.800819 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc-agent-certs\") pod \"konnectivity-agent-f9pjr\" (UID: \"9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc\") " pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:14.802488 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.802445 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8ede9a85-51fc-4f63-905f-659eeb168b3f-tmp\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.802592 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.802496 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8ede9a85-51fc-4f63-905f-659eeb168b3f-etc-tuned\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.804242 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.804204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-ovn-node-metrics-cert\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.814875 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.814852 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:14.814875 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.814875 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:14.815058 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.814884 2559 projected.go:196] Error preparing data for projected volume kube-api-access-x42tr for pod openshift-network-diagnostics/network-check-target-jr5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:14.815058 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:14.814955 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr podName:db40c4ce-12ac-41c2-9b56-6a53b5616259 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:15.314938092 +0000 UTC m=+3.114059936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x42tr" (UniqueName: "kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr") pod "network-check-target-jr5jk" (UID: "db40c4ce-12ac-41c2-9b56-6a53b5616259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:14.815187 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.815063 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8nv\" (UniqueName: \"kubernetes.io/projected/99e68b22-58b0-4760-9209-9732571d499c-kube-api-access-qr8nv\") pod \"aws-ebs-csi-driver-node-j2x6c\" (UID: \"99e68b22-58b0-4760-9209-9732571d499c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.816123 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.815482 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g78hx\" (UniqueName: \"kubernetes.io/projected/8ede9a85-51fc-4f63-905f-659eeb168b3f-kube-api-access-g78hx\") pod \"tuned-75g4k\" (UID: \"8ede9a85-51fc-4f63-905f-659eeb168b3f\") " pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.820109 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.820090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6mdj\" (UniqueName: \"kubernetes.io/projected/227b69aa-2d1b-4bf7-8603-f19fb4f2a606-kube-api-access-q6mdj\") pod \"node-ca-m9mfd\" (UID: \"227b69aa-2d1b-4bf7-8603-f19fb4f2a606\") " pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:14.820109 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.820104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85wq\" (UniqueName: \"kubernetes.io/projected/4fa27ea5-e7d8-4be1-9c00-410f85826d80-kube-api-access-f85wq\") pod \"multus-additional-cni-plugins-xvmpj\" (UID: \"4fa27ea5-e7d8-4be1-9c00-410f85826d80\") " pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.821371 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.820884 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fff2r\" (UniqueName: \"kubernetes.io/projected/1f6efc2c-af56-4997-92ec-93fbb8a3fb4e-kube-api-access-fff2r\") pod \"iptables-alerter-dzfmw\" (UID: \"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e\") " pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.822523 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.822499 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb5bj\" (UniqueName: \"kubernetes.io/projected/3198a50c-14d2-4981-8d5d-7718876d54b1-kube-api-access-xb5bj\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:14.823016 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.822968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgzx\" (UniqueName: \"kubernetes.io/projected/2aef5ab2-8f8b-4a39-a986-7e7fb777a934-kube-api-access-8hgzx\") pod \"ovnkube-node-v2qjb\" (UID: \"2aef5ab2-8f8b-4a39-a986-7e7fb777a934\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.893006 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.892956 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-57pgm" Sep 16 18:45:14.900550 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.900511 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9gp8g" Sep 16 18:45:14.912721 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:14.912508 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46efde69_5c9c_4fee_956d_f2c2c5855225.slice/crio-7bbddce4c53c6d22f53cb763f660f21896598ead3ca57b532bfc825885110a17 WatchSource:0}: Error finding container 7bbddce4c53c6d22f53cb763f660f21896598ead3ca57b532bfc825885110a17: Status 404 returned error can't find the container with id 7bbddce4c53c6d22f53cb763f660f21896598ead3ca57b532bfc825885110a17 Sep 16 18:45:14.915466 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.915443 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:14.923191 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.923167 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" Sep 16 18:45:14.926911 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:14.926888 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aef5ab2_8f8b_4a39_a986_7e7fb777a934.slice/crio-7ecae27108c027c443e68c73b7b02b84d0e9d75294a81075509892950f791734 WatchSource:0}: Error finding container 7ecae27108c027c443e68c73b7b02b84d0e9d75294a81075509892950f791734: Status 404 returned error can't find the container with id 7ecae27108c027c443e68c73b7b02b84d0e9d75294a81075509892950f791734 Sep 16 18:45:14.932114 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.932093 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" Sep 16 18:45:14.935289 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:14.935263 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e68b22_58b0_4760_9209_9732571d499c.slice/crio-4607086b8c891e06ef1c610607d6d0bfb15421d5305235b8e2667074834bfd92 WatchSource:0}: Error finding container 4607086b8c891e06ef1c610607d6d0bfb15421d5305235b8e2667074834bfd92: Status 404 returned error can't find the container with id 4607086b8c891e06ef1c610607d6d0bfb15421d5305235b8e2667074834bfd92 Sep 16 18:45:14.942143 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:14.942123 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fa27ea5_e7d8_4be1_9c00_410f85826d80.slice/crio-0c087f035d9ac03d19be73db5bd8ec9052f65557ffadbbf69b43cc2d2ca547c0 WatchSource:0}: Error finding container 0c087f035d9ac03d19be73db5bd8ec9052f65557ffadbbf69b43cc2d2ca547c0: Status 404 returned error can't find the container with id 0c087f035d9ac03d19be73db5bd8ec9052f65557ffadbbf69b43cc2d2ca547c0 Sep 16 18:45:14.948904 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.948881 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dzfmw" Sep 16 18:45:14.959527 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.959506 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:14.966510 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.966489 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-75g4k" Sep 16 18:45:14.973885 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:14.973864 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m9mfd" Sep 16 18:45:15.302002 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.301885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:15.302002 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.301925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:15.302240 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:15.302054 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:15.302240 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:15.302116 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs podName:3198a50c-14d2-4981-8d5d-7718876d54b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.302100898 +0000 UTC m=+4.101222718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs") pod "network-metrics-daemon-x7nkj" (UID: "3198a50c-14d2-4981-8d5d-7718876d54b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:15.302240 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:15.302054 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:15.302240 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:15.302197 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret podName:79ae6a44-0eb1-43cc-a9ff-63bc5342a979 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.302182495 +0000 UTC m=+4.101304336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret") pod "global-pull-secret-syncer-sz2r4" (UID: "79ae6a44-0eb1-43cc-a9ff-63bc5342a979") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:15.403157 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.403121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:15.403339 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:15.403272 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:15.403339 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:15.403294 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:15.403339 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:15.403305 2559 projected.go:196] Error preparing data for projected volume kube-api-access-x42tr for pod openshift-network-diagnostics/network-check-target-jr5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:15.403497 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:15.403364 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr podName:db40c4ce-12ac-41c2-9b56-6a53b5616259 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:16.403344685 +0000 UTC m=+4.202466504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42tr" (UniqueName: "kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr") pod "network-check-target-jr5jk" (UID: "db40c4ce-12ac-41c2-9b56-6a53b5616259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:15.663397 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.663317 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:15 +0000 UTC" deadline="2028-02-12 14:31:43.699354461 +0000 UTC" Sep 16 18:45:15.663397 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.663354 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12331h46m28.036004011s" Sep 16 18:45:15.730371 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.730328 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" event={"ID":"4fa27ea5-e7d8-4be1-9c00-410f85826d80","Type":"ContainerStarted","Data":"0c087f035d9ac03d19be73db5bd8ec9052f65557ffadbbf69b43cc2d2ca547c0"} Sep 16 18:45:15.733232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.733199 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" event={"ID":"99e68b22-58b0-4760-9209-9732571d499c","Type":"ContainerStarted","Data":"4607086b8c891e06ef1c610607d6d0bfb15421d5305235b8e2667074834bfd92"} Sep 16 18:45:15.736082 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.735953 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"7ecae27108c027c443e68c73b7b02b84d0e9d75294a81075509892950f791734"} Sep 16 18:45:15.738349 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.738326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9gp8g" event={"ID":"46efde69-5c9c-4fee-956d-f2c2c5855225","Type":"ContainerStarted","Data":"7bbddce4c53c6d22f53cb763f660f21896598ead3ca57b532bfc825885110a17"} Sep 16 18:45:15.740045 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:15.740014 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57pgm" event={"ID":"956fd3b6-79ea-4195-8ce8-35658bd3d7de","Type":"ContainerStarted","Data":"8152b3a88af1166a1593d40cbf85d1255ecb6c151d883cee7b4b54460b6ffc13"} Sep 16 18:45:16.311113 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.311072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:16.311298 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.311129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:16.311373 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.311301 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:16.311373 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.311368 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs podName:3198a50c-14d2-4981-8d5d-7718876d54b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:18.311348448 +0000 UTC m=+6.110470272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs") pod "network-metrics-daemon-x7nkj" (UID: "3198a50c-14d2-4981-8d5d-7718876d54b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:16.311882 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.311792 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:16.311882 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.311850 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret podName:79ae6a44-0eb1-43cc-a9ff-63bc5342a979 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:18.311834242 +0000 UTC m=+6.110956064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret") pod "global-pull-secret-syncer-sz2r4" (UID: "79ae6a44-0eb1-43cc-a9ff-63bc5342a979") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:16.411702 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.411665 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:16.411886 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.411868 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:16.411939 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.411893 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:16.411939 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.411906 2559 projected.go:196] Error preparing data for projected volume kube-api-access-x42tr for pod openshift-network-diagnostics/network-check-target-jr5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:16.412048 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.411979 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr podName:db40c4ce-12ac-41c2-9b56-6a53b5616259 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:18.411949182 +0000 UTC m=+6.211071026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42tr" (UniqueName: "kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr") pod "network-check-target-jr5jk" (UID: "db40c4ce-12ac-41c2-9b56-6a53b5616259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:16.427280 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:16.426617 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ede9a85_51fc_4f63_905f_659eeb168b3f.slice/crio-e8971baa6f2b7141fd6d3d47444e4647f341e488b3febfd2153c11594d02a42b WatchSource:0}: Error finding container e8971baa6f2b7141fd6d3d47444e4647f341e488b3febfd2153c11594d02a42b: Status 404 returned error can't find the container with id e8971baa6f2b7141fd6d3d47444e4647f341e488b3febfd2153c11594d02a42b Sep 16 18:45:16.453925 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:16.453895 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod227b69aa_2d1b_4bf7_8603_f19fb4f2a606.slice/crio-2c31f3a7bed3cce33a778304a574777f89193d99075af3d8bed316ffb2ffe4cd WatchSource:0}: Error finding container 2c31f3a7bed3cce33a778304a574777f89193d99075af3d8bed316ffb2ffe4cd: Status 404 returned error can't find the container with id 2c31f3a7bed3cce33a778304a574777f89193d99075af3d8bed316ffb2ffe4cd Sep 16 18:45:16.455005 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:16.454981 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6efc2c_af56_4997_92ec_93fbb8a3fb4e.slice/crio-ba18edb3d976868a65b8465b8452a9c4ef3664221902e0baa23ab7ed42fb2434 WatchSource:0}: Error finding container ba18edb3d976868a65b8465b8452a9c4ef3664221902e0baa23ab7ed42fb2434: Status 404 returned error can't find the container with id ba18edb3d976868a65b8465b8452a9c4ef3664221902e0baa23ab7ed42fb2434 Sep 16 18:45:16.664469 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.664402 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:15 +0000 UTC" deadline="2028-04-08 00:52:59.384705634 +0000 UTC" Sep 16 18:45:16.664469 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.664467 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13662h7m42.720244062s" Sep 16 18:45:16.717208 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.717176 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:16.717397 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.717321 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:16.717397 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.717345 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:16.717532 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.717476 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:16.717532 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.717179 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:16.717640 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:16.717575 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:16.746180 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.746140 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dzfmw" event={"ID":"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e","Type":"ContainerStarted","Data":"ba18edb3d976868a65b8465b8452a9c4ef3664221902e0baa23ab7ed42fb2434"} Sep 16 18:45:16.753220 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.753177 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f9pjr" event={"ID":"9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc","Type":"ContainerStarted","Data":"2eec56c25a1073dff6fa34c7e49786c297584138ae269005bfd9116e81dd8f83"} Sep 16 18:45:16.759383 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.759351 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-75g4k" event={"ID":"8ede9a85-51fc-4f63-905f-659eeb168b3f","Type":"ContainerStarted","Data":"e8971baa6f2b7141fd6d3d47444e4647f341e488b3febfd2153c11594d02a42b"} Sep 16 18:45:16.774918 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.774844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" event={"ID":"d512deac78b026258432227d6ed92f58","Type":"ContainerStarted","Data":"11ddc73dd7e1f8ac23e7580ac819080c65bf7167b3429797f609a2544a089ad3"} Sep 16 18:45:16.778662 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.778633 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m9mfd" event={"ID":"227b69aa-2d1b-4bf7-8603-f19fb4f2a606","Type":"ContainerStarted","Data":"2c31f3a7bed3cce33a778304a574777f89193d99075af3d8bed316ffb2ffe4cd"} Sep 16 18:45:16.802196 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:16.802135 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-230.ec2.internal" podStartSLOduration=2.80211647 podStartE2EDuration="2.80211647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:16.802001347 +0000 UTC m=+4.601123190" watchObservedRunningTime="2026-09-16 18:45:16.80211647 +0000 UTC m=+4.601238310" Sep 16 18:45:17.793472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:17.793431 2559 generic.go:356] "Generic (PLEG): container finished" podID="bb404389450f16abc78fd737f64d12f5" containerID="81a9735f6243055754b434127b9d8a0c816dc9a594d69b883485562a381a99f7" exitCode=0 Sep 16 18:45:17.794061 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:17.793511 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" event={"ID":"bb404389450f16abc78fd737f64d12f5","Type":"ContainerDied","Data":"81a9735f6243055754b434127b9d8a0c816dc9a594d69b883485562a381a99f7"} Sep 16 18:45:18.327263 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:18.327198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:18.327451 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:18.327305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:18.327535 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.327455 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:18.327535 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.327525 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs podName:3198a50c-14d2-4981-8d5d-7718876d54b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.32750397 +0000 UTC m=+10.126625793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs") pod "network-metrics-daemon-x7nkj" (UID: "3198a50c-14d2-4981-8d5d-7718876d54b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:18.327992 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.327969 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:18.328130 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.328027 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret podName:79ae6a44-0eb1-43cc-a9ff-63bc5342a979 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.328011001 +0000 UTC m=+10.127132823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret") pod "global-pull-secret-syncer-sz2r4" (UID: "79ae6a44-0eb1-43cc-a9ff-63bc5342a979") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:18.428893 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:18.428251 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:18.428893 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.428418 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:18.428893 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.428441 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:18.428893 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.428454 2559 projected.go:196] Error preparing data for projected volume kube-api-access-x42tr for pod openshift-network-diagnostics/network-check-target-jr5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:18.428893 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.428519 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr podName:db40c4ce-12ac-41c2-9b56-6a53b5616259 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:22.428497553 +0000 UTC m=+10.227619387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42tr" (UniqueName: "kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr") pod "network-check-target-jr5jk" (UID: "db40c4ce-12ac-41c2-9b56-6a53b5616259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:18.717253 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:18.717166 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:18.717253 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:18.717203 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:18.717463 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.717317 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:18.717463 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:18.717365 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:18.717587 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.717478 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:18.717587 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:18.717573 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:20.717781 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:20.717134 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:20.717781 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:20.717271 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:20.717781 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:20.717637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:20.717781 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:20.717725 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:20.718662 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:20.718502 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:20.718662 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:20.718620 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:22.364619 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:22.364587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:22.365164 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:22.364635 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:22.365164 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.364773 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:22.365164 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.364836 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs podName:3198a50c-14d2-4981-8d5d-7718876d54b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:30.364816181 +0000 UTC m=+18.163938010 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs") pod "network-metrics-daemon-x7nkj" (UID: "3198a50c-14d2-4981-8d5d-7718876d54b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:22.365336 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.365228 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:22.365336 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.365281 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret podName:79ae6a44-0eb1-43cc-a9ff-63bc5342a979 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:30.365265165 +0000 UTC m=+18.164386996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret") pod "global-pull-secret-syncer-sz2r4" (UID: "79ae6a44-0eb1-43cc-a9ff-63bc5342a979") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:22.465887 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:22.465850 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:22.466079 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.466003 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:22.466079 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.466023 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:22.466079 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.466036 2559 projected.go:196] Error preparing data for projected volume kube-api-access-x42tr for pod openshift-network-diagnostics/network-check-target-jr5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:22.466229 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.466089 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr podName:db40c4ce-12ac-41c2-9b56-6a53b5616259 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:30.466072983 +0000 UTC m=+18.265194813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42tr" (UniqueName: "kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr") pod "network-check-target-jr5jk" (UID: "db40c4ce-12ac-41c2-9b56-6a53b5616259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:22.718014 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:22.717199 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:22.718014 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.717312 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:22.718014 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:22.717703 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:22.718014 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.717801 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:22.718014 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:22.717852 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:22.718014 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:22.717918 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:24.716232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:24.716194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:24.716667 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:24.716194 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:24.716667 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:24.716316 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:24.716667 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:24.716401 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:24.716667 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:24.716204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:24.716667 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:24.716515 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:26.716262 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:26.716092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:26.716262 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:26.716147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:26.716262 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:26.716107 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:26.716262 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:26.716234 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:26.716885 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:26.716300 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:26.716885 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:26.716367 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:28.720070 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:28.720037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:28.720528 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:28.720086 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:28.720528 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:28.720037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:28.720528 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:28.720164 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:28.720528 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:28.720257 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:28.720528 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:28.720356 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:30.428974 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:30.428935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:30.429412 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:30.429029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:30.429412 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.429093 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:30.429412 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.429162 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:30.429412 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.429170 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret podName:79ae6a44-0eb1-43cc-a9ff-63bc5342a979 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.429151031 +0000 UTC m=+34.228272869 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret") pod "global-pull-secret-syncer-sz2r4" (UID: "79ae6a44-0eb1-43cc-a9ff-63bc5342a979") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:30.429412 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.429200 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs podName:3198a50c-14d2-4981-8d5d-7718876d54b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.429188955 +0000 UTC m=+34.228310784 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs") pod "network-metrics-daemon-x7nkj" (UID: "3198a50c-14d2-4981-8d5d-7718876d54b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:30.530083 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:30.530049 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:30.530243 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.530208 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:30.530243 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.530231 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:30.530243 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.530242 2559 projected.go:196] Error preparing data for projected volume kube-api-access-x42tr for pod openshift-network-diagnostics/network-check-target-jr5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:30.530400 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.530312 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr podName:db40c4ce-12ac-41c2-9b56-6a53b5616259 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:46.530291442 +0000 UTC m=+34.329413261 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42tr" (UniqueName: "kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr") pod "network-check-target-jr5jk" (UID: "db40c4ce-12ac-41c2-9b56-6a53b5616259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:30.719984 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:30.719898 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:30.720137 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:30.719894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:30.720137 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.720023 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:30.720137 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:30.719894 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:30.720137 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.720105 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:30.720342 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:30.720172 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:32.716894 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:32.716862 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:32.717361 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:32.716905 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:32.717361 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:32.716975 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:32.717361 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:32.717034 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:32.717361 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:32.717097 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:32.717361 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:32.717198 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:34.719337 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.719187 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:34.719939 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.719258 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:34.719939 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:34.719407 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:34.719939 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:34.719500 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:34.719939 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.719258 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:34.719939 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:34.719634 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:34.830988 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.830895 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m9mfd" event={"ID":"227b69aa-2d1b-4bf7-8603-f19fb4f2a606","Type":"ContainerStarted","Data":"6ae789dd1e0144f0de2a2831f2ea859095fb334c9ad0372991c836b0ba980dd0"} Sep 16 18:45:34.833050 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.833023 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" event={"ID":"bb404389450f16abc78fd737f64d12f5","Type":"ContainerStarted","Data":"df8a3c85ea5a4f36e6c4cbe749336a59c49301a61b1eda9f1c8192a43933f3b2"} Sep 16 18:45:34.834704 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.834677 2559 generic.go:356] "Generic (PLEG): container finished" podID="4fa27ea5-e7d8-4be1-9c00-410f85826d80" containerID="b5e8a3c65ecfee23e66d3b4e080bd6f316fa3382ab349936720f6c68e0e80262" exitCode=0 Sep 16 18:45:34.834807 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.834763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" event={"ID":"4fa27ea5-e7d8-4be1-9c00-410f85826d80","Type":"ContainerDied","Data":"b5e8a3c65ecfee23e66d3b4e080bd6f316fa3382ab349936720f6c68e0e80262"} Sep 16 18:45:34.836661 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.836639 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" event={"ID":"99e68b22-58b0-4760-9209-9732571d499c","Type":"ContainerStarted","Data":"396fba6d808fb4adec993cb9098fce2faf09003302c29ab19eeb691663e684a5"} Sep 16 18:45:34.838428 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.838407 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9gp8g" event={"ID":"46efde69-5c9c-4fee-956d-f2c2c5855225","Type":"ContainerStarted","Data":"06f7024675cc8452a0be83b86998eb9829c461df0f035e077a5e95955f25367b"} Sep 16 18:45:34.840058 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.840031 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-57pgm" event={"ID":"956fd3b6-79ea-4195-8ce8-35658bd3d7de","Type":"ContainerStarted","Data":"c9d69b0bc610fb9c6914f730230c908e8677e2c242fdcf35d0c93ac8cb15461e"} Sep 16 18:45:34.841666 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.841643 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f9pjr" event={"ID":"9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc","Type":"ContainerStarted","Data":"896dff7927b44ea594e232e883b64bcac983970de12f0798d5cf3c962d835457"} Sep 16 18:45:34.843322 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.843302 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-75g4k" event={"ID":"8ede9a85-51fc-4f63-905f-659eeb168b3f","Type":"ContainerStarted","Data":"db9d56e7290441881aefbe09b9c41c1ce4065b80047f0335965e347ee16b3346"} Sep 16 18:45:34.846450 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.846433 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 18:45:34.846834 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.846810 2559 generic.go:356] "Generic (PLEG): container finished" podID="2aef5ab2-8f8b-4a39-a986-7e7fb777a934" containerID="4237f2ec10cfdeeb3ecbe60e7d7feee9826b4dd1ae7060caf09668c08f68bbe0" exitCode=1 Sep 16 18:45:34.846918 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.846839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"fae56c0f3c5e54721231e057c85ee48add6749a16e04a408b784ba5159918805"} Sep 16 18:45:34.846995 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.846916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"0d0f3f6eaf2067e5be3158a574e820eaf85e4a22cf7e5130f74795607b9cd4fe"} Sep 16 18:45:34.846995 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.846983 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"8ccf2df7117c8b564b0b6cea2309b59cc07c389cbb0e59d128b71acae34c22bb"} Sep 16 18:45:34.847090 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.847036 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"2f3aed564848fe83401837be6ce0f74af6371e494394382f674fc94a3bd247e0"} Sep 16 18:45:34.847151 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.847085 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerDied","Data":"4237f2ec10cfdeeb3ecbe60e7d7feee9826b4dd1ae7060caf09668c08f68bbe0"} Sep 16 18:45:34.847151 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.847137 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"30879d8b88fab54ea315e658d28de6aa8d86012a39c66ab9196b1e5b9d87c742"} Sep 16 18:45:34.868047 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.867999 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m9mfd" podStartSLOduration=4.585547143 podStartE2EDuration="21.867985442s" totalImagesPullingTime="17.282438299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.850507002 +0000 UTC m=+22.649628851" watchObservedRunningTime="2026-09-16 18:45:34.867985442 +0000 UTC m=+22.667107283" Sep 16 18:45:34.894928 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.894879 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f9pjr" podStartSLOduration=4.631672314 podStartE2EDuration="21.894864902s" totalImagesPullingTime="17.263192588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.868161784 +0000 UTC m=+22.667283645" watchObservedRunningTime="2026-09-16 18:45:34.894864902 +0000 UTC m=+22.693986740" Sep 16 18:45:34.912349 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.912298 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-75g4k" podStartSLOduration=4.597304341 podStartE2EDuration="21.912280116s" totalImagesPullingTime="17.314975775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.894411128 +0000 UTC m=+22.693532968" watchObservedRunningTime="2026-09-16 18:45:34.912280116 +0000 UTC m=+22.711401996" Sep 16 18:45:34.912507 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.912472 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-230.ec2.internal" podStartSLOduration=20.912466592 podStartE2EDuration="20.912466592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.91239197 +0000 UTC m=+22.711513834" watchObservedRunningTime="2026-09-16 18:45:34.912466592 +0000 UTC m=+22.711588431" Sep 16 18:45:34.935722 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.935677 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-57pgm" podStartSLOduration=3.074373644 podStartE2EDuration="21.93565933s" totalImagesPullingTime="18.861285686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.935141415 +0000 UTC m=+22.734263274" watchObservedRunningTime="2026-09-16 18:45:34.93565933 +0000 UTC m=+22.734781169" Sep 16 18:45:34.979848 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:34.979798 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9gp8g" podStartSLOduration=2.7008004789999998 podStartE2EDuration="21.979785095s" totalImagesPullingTime="19.278984616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:34.979479808 +0000 UTC m=+22.778601667" watchObservedRunningTime="2026-09-16 18:45:34.979785095 +0000 UTC m=+22.778906934" Sep 16 18:45:35.102364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:35.102336 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 18:45:35.643808 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:35.643550 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:35.102351822Z","UUID":"2ce7ce45-31c4-4c7b-b955-09ac8c481c29","Handler":null,"Name":"","Endpoint":""} Sep 16 18:45:35.645138 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:35.645112 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 18:45:35.645265 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:35.645152 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 18:45:35.851464 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:35.851242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dzfmw" event={"ID":"1f6efc2c-af56-4997-92ec-93fbb8a3fb4e","Type":"ContainerStarted","Data":"3088e7c478c050de7a643974d3b62d49de5a69fa22b5d70bca7a002a73d54158"} Sep 16 18:45:35.854225 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:35.854146 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" event={"ID":"99e68b22-58b0-4760-9209-9732571d499c","Type":"ContainerStarted","Data":"f5638e9db52d2bea4997ff14c58ea8fcb3fcbc12ddb37afc49b30485a85f4ea1"} Sep 16 18:45:35.875880 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:35.875823 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dzfmw" podStartSLOduration=5.571716013 podStartE2EDuration="22.875799471s" totalImagesPullingTime="17.304083458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:35.875465139 +0000 UTC m=+23.674587258" watchObservedRunningTime="2026-09-16 18:45:35.875799471 +0000 UTC m=+23.674921311" Sep 16 18:45:36.715912 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:36.715839 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:36.716105 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:36.715840 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:36.716105 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:36.715955 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:36.716105 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:36.715843 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:36.716105 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:36.716041 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:36.716306 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:36.716142 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:36.860382 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:36.860349 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 18:45:36.860854 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:36.860833 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"a6e76a2bfc7f52b0c64a4fceb35c75fadf22e811e3bc94f100759367549e333d"} Sep 16 18:45:36.863216 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:36.863184 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" event={"ID":"99e68b22-58b0-4760-9209-9732571d499c","Type":"ContainerStarted","Data":"8401f27f6aab1b211fde338ff268ff485a62405cdf55d5ebdf31c5f89ab2932e"} Sep 16 18:45:36.887104 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:36.886998 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j2x6c" podStartSLOduration=2.826617401 podStartE2EDuration="23.88695893s" totalImagesPullingTime="21.060341529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:36.886062372 +0000 UTC m=+24.685184231" watchObservedRunningTime="2026-09-16 18:45:36.88695893 +0000 UTC m=+24.686080769" Sep 16 18:45:37.297283 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:37.297245 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:37.297861 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:37.297839 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:38.294149 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.294110 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:38.294612 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.294595 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f9pjr" Sep 16 18:45:38.715982 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.715940 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:38.716171 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.715949 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:38.716171 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:38.716063 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:38.716171 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.715950 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:38.716330 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:38.716181 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:38.716330 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:38.716258 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:38.871697 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.871556 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 18:45:38.872143 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.872123 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"786edc7c9d6d49d8f6d7cb10c22892c7cdc5234aae57934b8f67cd00e2f95091"} Sep 16 18:45:38.872512 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.872489 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:38.872585 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.872559 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:38.872649 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.872591 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:38.872702 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.872654 2559 scope.go:117] "RemoveContainer" containerID="4237f2ec10cfdeeb3ecbe60e7d7feee9826b4dd1ae7060caf09668c08f68bbe0" Sep 16 18:45:38.889671 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.889652 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:38.893546 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:38.893459 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:45:39.878105 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:39.878078 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 18:45:39.878634 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:39.878423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" event={"ID":"2aef5ab2-8f8b-4a39-a986-7e7fb777a934","Type":"ContainerStarted","Data":"06a876ed8c6413bdc08d19a940e2ecec692565977b2fa59f60bee0ca0b432d67"} Sep 16 18:45:39.880211 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:39.880180 2559 generic.go:356] "Generic (PLEG): container finished" podID="4fa27ea5-e7d8-4be1-9c00-410f85826d80" containerID="1fe00d7035e1dbdffc05156ac6fdaf58ce504617c1f5417d021b0b62cf299ad6" exitCode=0 Sep 16 18:45:39.880320 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:39.880262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" event={"ID":"4fa27ea5-e7d8-4be1-9c00-410f85826d80","Type":"ContainerDied","Data":"1fe00d7035e1dbdffc05156ac6fdaf58ce504617c1f5417d021b0b62cf299ad6"} Sep 16 18:45:39.957840 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:39.957786 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" podStartSLOduration=8.104809651 podStartE2EDuration="26.957765996s" totalImagesPullingTime="18.852956345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:39.92141626 +0000 UTC m=+27.720538120" watchObservedRunningTime="2026-09-16 18:45:39.957765996 +0000 UTC m=+27.756887837" Sep 16 18:45:40.716560 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.716530 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:40.716560 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.716543 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:40.716805 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.716591 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:40.716805 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:40.716694 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:40.716805 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:40.716770 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:40.716941 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:40.716840 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:40.817818 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.817787 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jr5jk"] Sep 16 18:45:40.820321 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.820299 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sz2r4"] Sep 16 18:45:40.837409 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.837381 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7nkj"] Sep 16 18:45:40.882617 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.882597 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:40.883259 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.882603 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:40.883259 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:40.883039 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:40.883259 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:40.883124 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:40.883259 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:40.883217 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:40.883465 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:40.883288 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:41.886680 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:41.886641 2559 generic.go:356] "Generic (PLEG): container finished" podID="4fa27ea5-e7d8-4be1-9c00-410f85826d80" containerID="aecedac984916597756e41abb140981a713922c9bc4633374a1930e4f2982eae" exitCode=0 Sep 16 18:45:41.887154 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:41.886703 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" event={"ID":"4fa27ea5-e7d8-4be1-9c00-410f85826d80","Type":"ContainerDied","Data":"aecedac984916597756e41abb140981a713922c9bc4633374a1930e4f2982eae"} Sep 16 18:45:42.716852 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:42.716829 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:42.716989 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:42.716938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:42.716989 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:42.716945 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:42.717174 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:42.717063 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:42.717174 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:42.717098 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:42.717257 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:42.717187 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:42.891102 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:42.891070 2559 generic.go:356] "Generic (PLEG): container finished" podID="4fa27ea5-e7d8-4be1-9c00-410f85826d80" containerID="4133c2d03c519aaf42c3d204dacdf7011ef8382935902620a2b61d31e87e0fcc" exitCode=0 Sep 16 18:45:42.891446 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:42.891118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" event={"ID":"4fa27ea5-e7d8-4be1-9c00-410f85826d80","Type":"ContainerDied","Data":"4133c2d03c519aaf42c3d204dacdf7011ef8382935902620a2b61d31e87e0fcc"} Sep 16 18:45:44.717171 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:44.716922 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:44.717589 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:44.716925 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:44.717589 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:44.717278 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:44.717589 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:44.716996 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:44.717589 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:44.717356 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:44.717589 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:44.717551 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:46.444535 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:46.444503 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:46.445122 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:46.444543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:46.445122 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.444647 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:46.445122 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.444662 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:46.445122 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.444702 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs podName:3198a50c-14d2-4981-8d5d-7718876d54b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.444685422 +0000 UTC m=+66.243807255 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs") pod "network-metrics-daemon-x7nkj" (UID: "3198a50c-14d2-4981-8d5d-7718876d54b1") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:46.445122 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.444717 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret podName:79ae6a44-0eb1-43cc-a9ff-63bc5342a979 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.444710977 +0000 UTC m=+66.243832794 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret") pod "global-pull-secret-syncer-sz2r4" (UID: "79ae6a44-0eb1-43cc-a9ff-63bc5342a979") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:46.545748 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:46.545716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:46.545923 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.545890 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:46.545923 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.545912 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:46.545923 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.545921 2559 projected.go:196] Error preparing data for projected volume kube-api-access-x42tr for pod openshift-network-diagnostics/network-check-target-jr5jk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:46.546091 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.545988 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr podName:db40c4ce-12ac-41c2-9b56-6a53b5616259 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.545958572 +0000 UTC m=+66.345080408 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x42tr" (UniqueName: "kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr") pod "network-check-target-jr5jk" (UID: "db40c4ce-12ac-41c2-9b56-6a53b5616259") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:46.716029 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:46.715936 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:46.716029 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:46.715988 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:46.716265 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:46.715940 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:46.716265 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.716102 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sz2r4" podUID="79ae6a44-0eb1-43cc-a9ff-63bc5342a979" Sep 16 18:45:46.716265 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.716212 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x7nkj" podUID="3198a50c-14d2-4981-8d5d-7718876d54b1" Sep 16 18:45:46.716420 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:46.716301 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jr5jk" podUID="db40c4ce-12ac-41c2-9b56-6a53b5616259" Sep 16 18:45:47.024896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.024827 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-230.ec2.internal" event="NodeReady" Sep 16 18:45:47.025073 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.024979 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 18:45:47.068200 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.068171 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d455dc6cd-bp6wv"] Sep 16 18:45:47.082149 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.082121 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp"] Sep 16 18:45:47.082280 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.082240 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.085169 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.085150 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-htp7b\"" Sep 16 18:45:47.085433 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.085420 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 18:45:47.085691 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.085678 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 18:45:47.085745 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.085693 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 18:45:47.091655 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.091630 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 18:45:47.106318 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.106289 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp"] Sep 16 18:45:47.106433 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.106410 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" Sep 16 18:45:47.109558 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.109540 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-72pz8\"" Sep 16 18:45:47.109901 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.109886 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 18:45:47.109998 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.109908 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 18:45:47.110259 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.110245 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 18:45:47.110480 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.110468 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 16 18:45:47.118670 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.118650 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2"] Sep 16 18:45:47.118803 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.118787 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.121633 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.121618 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 16 18:45:47.132744 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.132729 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp"] Sep 16 18:45:47.132840 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.132758 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp"] Sep 16 18:45:47.132840 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.132770 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hmp8z"] Sep 16 18:45:47.132916 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.132867 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.135846 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.135682 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 16 18:45:47.135846 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.135724 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 16 18:45:47.135846 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.135742 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 16 18:45:47.136081 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.135983 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 16 18:45:47.146887 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.146869 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cq5vr"] Sep 16 18:45:47.147010 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.146997 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.150552 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.150532 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 18:45:47.150629 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.150589 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 18:45:47.150708 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.150693 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8559\"" Sep 16 18:45:47.167075 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.167057 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2"] Sep 16 18:45:47.167151 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.167083 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d455dc6cd-bp6wv"] Sep 16 18:45:47.167151 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.167097 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmp8z"] Sep 16 18:45:47.167151 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.167117 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cq5vr"] Sep 16 18:45:47.167242 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.167185 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cq5vr" Sep 16 18:45:47.170697 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.170679 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 18:45:47.171800 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.171785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dts4n\"" Sep 16 18:45:47.171857 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.171812 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 18:45:47.171903 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.171870 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 18:45:47.251173 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.251173 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e430d3bb-7ea2-49f9-819b-5211dc39a851-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.251354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251206 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.251354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251221 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c555115e-10aa-47fb-8e27-6dd97a9b769c-tmp\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.251354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-image-registry-private-configuration\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.251354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-ca\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.251488 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-ca-trust-extracted\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.251488 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251391 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6zn2\" (UniqueName: \"kubernetes.io/projected/e0e4c7f6-27c5-4b24-b948-19547c652307-kube-api-access-d6zn2\") pod \"managed-serviceaccount-addon-agent-69d9d7484d-bt9zp\" (UID: \"e0e4c7f6-27c5-4b24-b948-19547c652307\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" Sep 16 18:45:47.251488 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251421 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-certificates\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.251488 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-bound-sa-token\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.251488 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251466 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvv6x\" (UniqueName: \"kubernetes.io/projected/c555115e-10aa-47fb-8e27-6dd97a9b769c-kube-api-access-kvv6x\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251481 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-trusted-ca\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8h7\" (UniqueName: \"kubernetes.io/projected/e430d3bb-7ea2-49f9-819b-5211dc39a851-kube-api-access-pf8h7\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251567 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0e4c7f6-27c5-4b24-b948-19547c652307-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9d7484d-bt9zp\" (UID: \"e0e4c7f6-27c5-4b24-b948-19547c652307\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251580 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c555115e-10aa-47fb-8e27-6dd97a9b769c-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-hub\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251607 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxjvr\" (UniqueName: \"kubernetes.io/projected/9adce71e-476d-4ff2-9cd0-ce06e831d479-kube-api-access-nxjvr\") pod \"ingress-canary-cq5vr\" (UID: \"9adce71e-476d-4ff2-9cd0-ce06e831d479\") " pod="openshift-ingress-canary/ingress-canary-cq5vr" Sep 16 18:45:47.251673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-installation-pull-secrets\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.251925 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251692 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9adce71e-476d-4ff2-9cd0-ce06e831d479-cert\") pod \"ingress-canary-cq5vr\" (UID: \"9adce71e-476d-4ff2-9cd0-ce06e831d479\") " pod="openshift-ingress-canary/ingress-canary-cq5vr" Sep 16 18:45:47.251925 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251718 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkpkk\" (UniqueName: \"kubernetes.io/projected/aed88e62-448f-471e-8680-79d5d962b285-kube-api-access-dkpkk\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.251925 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251738 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztsnc\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-kube-api-access-ztsnc\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.251925 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aed88e62-448f-471e-8680-79d5d962b285-config-volume\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.251925 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.251796 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aed88e62-448f-471e-8680-79d5d962b285-tmp-dir\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.352532 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztsnc\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-kube-api-access-ztsnc\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.352532 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352514 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aed88e62-448f-471e-8680-79d5d962b285-config-volume\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.352730 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aed88e62-448f-471e-8680-79d5d962b285-tmp-dir\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.352730 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.352730 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e430d3bb-7ea2-49f9-819b-5211dc39a851-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.352730 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.352912 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.352735 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:47.352912 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.352748 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d455dc6cd-bp6wv: secret "image-registry-tls" not found Sep 16 18:45:47.352912 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.352820 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls podName:5f9e864f-101e-42ec-98bd-6b2051eb5fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.852802282 +0000 UTC m=+35.651924112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls") pod "image-registry-d455dc6cd-bp6wv" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6") : secret "image-registry-tls" not found Sep 16 18:45:47.352912 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c555115e-10aa-47fb-8e27-6dd97a9b769c-tmp\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.352912 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-image-registry-private-configuration\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.352912 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352911 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-ca\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352922 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aed88e62-448f-471e-8680-79d5d962b285-tmp-dir\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.352950 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-ca-trust-extracted\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6zn2\" (UniqueName: \"kubernetes.io/projected/e0e4c7f6-27c5-4b24-b948-19547c652307-kube-api-access-d6zn2\") pod \"managed-serviceaccount-addon-agent-69d9d7484d-bt9zp\" (UID: \"e0e4c7f6-27c5-4b24-b948-19547c652307\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-certificates\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-bound-sa-token\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvv6x\" (UniqueName: \"kubernetes.io/projected/c555115e-10aa-47fb-8e27-6dd97a9b769c-kube-api-access-kvv6x\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353140 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-trusted-ca\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353167 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8h7\" (UniqueName: \"kubernetes.io/projected/e430d3bb-7ea2-49f9-819b-5211dc39a851-kube-api-access-pf8h7\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.353232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0e4c7f6-27c5-4b24-b948-19547c652307-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9d7484d-bt9zp\" (UID: \"e0e4c7f6-27c5-4b24-b948-19547c652307\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353241 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c555115e-10aa-47fb-8e27-6dd97a9b769c-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c555115e-10aa-47fb-8e27-6dd97a9b769c-tmp\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-hub\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxjvr\" (UniqueName: \"kubernetes.io/projected/9adce71e-476d-4ff2-9cd0-ce06e831d479-kube-api-access-nxjvr\") pod \"ingress-canary-cq5vr\" (UID: \"9adce71e-476d-4ff2-9cd0-ce06e831d479\") " pod="openshift-ingress-canary/ingress-canary-cq5vr" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353349 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-installation-pull-secrets\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353375 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9adce71e-476d-4ff2-9cd0-ce06e831d479-cert\") pod \"ingress-canary-cq5vr\" (UID: \"9adce71e-476d-4ff2-9cd0-ce06e831d479\") " pod="openshift-ingress-canary/ingress-canary-cq5vr" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353380 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/aed88e62-448f-471e-8680-79d5d962b285-config-volume\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkpkk\" (UniqueName: \"kubernetes.io/projected/aed88e62-448f-471e-8680-79d5d962b285-kube-api-access-dkpkk\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.353517 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:47.353766 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.353567 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls podName:aed88e62-448f-471e-8680-79d5d962b285 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.853551372 +0000 UTC m=+35.652673203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls") pod "dns-default-hmp8z" (UID: "aed88e62-448f-471e-8680-79d5d962b285") : secret "dns-default-metrics-tls" not found Sep 16 18:45:47.354208 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.353988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e430d3bb-7ea2-49f9-819b-5211dc39a851-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.354208 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.354022 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-ca-trust-extracted\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.354314 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.354220 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-certificates\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.355018 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.354947 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-trusted-ca\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.358355 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.358330 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-image-registry-private-configuration\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.358458 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.358359 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.358458 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.358401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9adce71e-476d-4ff2-9cd0-ce06e831d479-cert\") pod \"ingress-canary-cq5vr\" (UID: \"9adce71e-476d-4ff2-9cd0-ce06e831d479\") " pod="openshift-ingress-canary/ingress-canary-cq5vr" Sep 16 18:45:47.358583 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.358564 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-installation-pull-secrets\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.358886 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.358862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-ca\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.359049 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.359029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c555115e-10aa-47fb-8e27-6dd97a9b769c-klusterlet-config\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.359121 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.359107 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e0e4c7f6-27c5-4b24-b948-19547c652307-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-69d9d7484d-bt9zp\" (UID: \"e0e4c7f6-27c5-4b24-b948-19547c652307\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" Sep 16 18:45:47.359620 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.359594 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.360020 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.360003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e430d3bb-7ea2-49f9-819b-5211dc39a851-hub\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.363505 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.363480 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztsnc\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-kube-api-access-ztsnc\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.363613 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.363575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6zn2\" (UniqueName: \"kubernetes.io/projected/e0e4c7f6-27c5-4b24-b948-19547c652307-kube-api-access-d6zn2\") pod \"managed-serviceaccount-addon-agent-69d9d7484d-bt9zp\" (UID: \"e0e4c7f6-27c5-4b24-b948-19547c652307\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" Sep 16 18:45:47.365017 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.364995 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-bound-sa-token\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.365907 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.365883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxjvr\" (UniqueName: \"kubernetes.io/projected/9adce71e-476d-4ff2-9cd0-ce06e831d479-kube-api-access-nxjvr\") pod \"ingress-canary-cq5vr\" (UID: \"9adce71e-476d-4ff2-9cd0-ce06e831d479\") " pod="openshift-ingress-canary/ingress-canary-cq5vr" Sep 16 18:45:47.366148 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.366129 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8h7\" (UniqueName: \"kubernetes.io/projected/e430d3bb-7ea2-49f9-819b-5211dc39a851-kube-api-access-pf8h7\") pod \"cluster-proxy-proxy-agent-d468b69db-lw6t2\" (UID: \"e430d3bb-7ea2-49f9-819b-5211dc39a851\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.366433 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.366418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvv6x\" (UniqueName: \"kubernetes.io/projected/c555115e-10aa-47fb-8e27-6dd97a9b769c-kube-api-access-kvv6x\") pod \"klusterlet-addon-workmgr-64cd77df94-hhckp\" (UID: \"c555115e-10aa-47fb-8e27-6dd97a9b769c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.366930 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.366915 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkpkk\" (UniqueName: \"kubernetes.io/projected/aed88e62-448f-471e-8680-79d5d962b285-kube-api-access-dkpkk\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.430920 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.430900 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" Sep 16 18:45:47.439112 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.439087 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:47.444249 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.444227 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:45:47.474616 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.474598 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cq5vr" Sep 16 18:45:47.515738 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.515578 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:47.648678 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.648647 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:47.760145 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.760118 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp"] Sep 16 18:45:47.762398 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.762377 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp"] Sep 16 18:45:47.804359 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.804332 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cq5vr"] Sep 16 18:45:47.827905 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.827881 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2"] Sep 16 18:45:47.830842 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:47.830820 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0e4c7f6_27c5_4b24_b948_19547c652307.slice/crio-f811e571a3f138778dc3efdce992026ed5ab1eeb231b3390fcc3341bf3b3e6ca WatchSource:0}: Error finding container f811e571a3f138778dc3efdce992026ed5ab1eeb231b3390fcc3341bf3b3e6ca: Status 404 returned error can't find the container with id f811e571a3f138778dc3efdce992026ed5ab1eeb231b3390fcc3341bf3b3e6ca Sep 16 18:45:47.831814 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:47.831787 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adce71e_476d_4ff2_9cd0_ce06e831d479.slice/crio-6c0702bcadaac0c0433b0d1636b9910ce936c705f6075af78fbc4ae528ed1fd3 WatchSource:0}: Error finding container 6c0702bcadaac0c0433b0d1636b9910ce936c705f6075af78fbc4ae528ed1fd3: Status 404 returned error can't find the container with id 6c0702bcadaac0c0433b0d1636b9910ce936c705f6075af78fbc4ae528ed1fd3 Sep 16 18:45:47.832920 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:47.832871 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc555115e_10aa_47fb_8e27_6dd97a9b769c.slice/crio-4a958eaf164ac02aecb46d8d158540cb20fa0305c8d0d48c090874b6963c95be WatchSource:0}: Error finding container 4a958eaf164ac02aecb46d8d158540cb20fa0305c8d0d48c090874b6963c95be: Status 404 returned error can't find the container with id 4a958eaf164ac02aecb46d8d158540cb20fa0305c8d0d48c090874b6963c95be Sep 16 18:45:47.834220 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:45:47.833693 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode430d3bb_7ea2_49f9_819b_5211dc39a851.slice/crio-ef8e65dee70ba28babab52d9072218ec9d0d5c197a48c38bb3ba5484325c23b7 WatchSource:0}: Error finding container ef8e65dee70ba28babab52d9072218ec9d0d5c197a48c38bb3ba5484325c23b7: Status 404 returned error can't find the container with id ef8e65dee70ba28babab52d9072218ec9d0d5c197a48c38bb3ba5484325c23b7 Sep 16 18:45:47.857932 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.857909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:47.858066 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.857982 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:47.858126 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.858084 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:47.858126 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.858107 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d455dc6cd-bp6wv: secret "image-registry-tls" not found Sep 16 18:45:47.858216 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.858090 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:47.858216 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.858170 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls podName:5f9e864f-101e-42ec-98bd-6b2051eb5fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.858148261 +0000 UTC m=+36.657270099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls") pod "image-registry-d455dc6cd-bp6wv" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6") : secret "image-registry-tls" not found Sep 16 18:45:47.858216 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:47.858213 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls podName:aed88e62-448f-471e-8680-79d5d962b285 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:48.858199722 +0000 UTC m=+36.657321540 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls") pod "dns-default-hmp8z" (UID: "aed88e62-448f-471e-8680-79d5d962b285") : secret "dns-default-metrics-tls" not found Sep 16 18:45:47.905292 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.905227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" event={"ID":"e430d3bb-7ea2-49f9-819b-5211dc39a851","Type":"ContainerStarted","Data":"ef8e65dee70ba28babab52d9072218ec9d0d5c197a48c38bb3ba5484325c23b7"} Sep 16 18:45:47.906287 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.906260 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" event={"ID":"c555115e-10aa-47fb-8e27-6dd97a9b769c","Type":"ContainerStarted","Data":"4a958eaf164ac02aecb46d8d158540cb20fa0305c8d0d48c090874b6963c95be"} Sep 16 18:45:47.907343 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.907319 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" event={"ID":"e0e4c7f6-27c5-4b24-b948-19547c652307","Type":"ContainerStarted","Data":"f811e571a3f138778dc3efdce992026ed5ab1eeb231b3390fcc3341bf3b3e6ca"} Sep 16 18:45:47.910000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.909976 2559 generic.go:356] "Generic (PLEG): container finished" podID="4fa27ea5-e7d8-4be1-9c00-410f85826d80" containerID="7ab901280f579a8296bc0ac0f3b5223ac8aafa937652ff2b7171ec18186be0aa" exitCode=0 Sep 16 18:45:47.910096 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.910052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" event={"ID":"4fa27ea5-e7d8-4be1-9c00-410f85826d80","Type":"ContainerDied","Data":"7ab901280f579a8296bc0ac0f3b5223ac8aafa937652ff2b7171ec18186be0aa"} Sep 16 18:45:47.911239 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:47.911217 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cq5vr" event={"ID":"9adce71e-476d-4ff2-9cd0-ce06e831d479","Type":"ContainerStarted","Data":"6c0702bcadaac0c0433b0d1636b9910ce936c705f6075af78fbc4ae528ed1fd3"} Sep 16 18:45:48.716578 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.716492 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:45:48.717470 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.716495 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:45:48.717470 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.716547 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:45:48.721843 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.721820 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtjvb\"" Sep 16 18:45:48.722034 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.721875 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqjhk\"" Sep 16 18:45:48.722034 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.721950 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:45:48.722195 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.721820 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:45:48.722195 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.722133 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:45:48.722195 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.722156 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:45:48.867822 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.867778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:48.868006 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.867910 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:48.868193 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:48.868175 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:48.868259 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:48.868227 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d455dc6cd-bp6wv: secret "image-registry-tls" not found Sep 16 18:45:48.868355 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:48.868337 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls podName:5f9e864f-101e-42ec-98bd-6b2051eb5fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.8683174 +0000 UTC m=+38.667439222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls") pod "image-registry-d455dc6cd-bp6wv" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6") : secret "image-registry-tls" not found Sep 16 18:45:48.868866 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:48.868846 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:48.868957 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:48.868900 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls podName:aed88e62-448f-471e-8680-79d5d962b285 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.868885749 +0000 UTC m=+38.668007572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls") pod "dns-default-hmp8z" (UID: "aed88e62-448f-471e-8680-79d5d962b285") : secret "dns-default-metrics-tls" not found Sep 16 18:45:48.925644 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.925608 2559 generic.go:356] "Generic (PLEG): container finished" podID="4fa27ea5-e7d8-4be1-9c00-410f85826d80" containerID="02d6dde10eb5d28d38eabb5d2506b7701fb2c545fce52b6908ea07459bcf8d33" exitCode=0 Sep 16 18:45:48.925804 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:48.925689 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" event={"ID":"4fa27ea5-e7d8-4be1-9c00-410f85826d80","Type":"ContainerDied","Data":"02d6dde10eb5d28d38eabb5d2506b7701fb2c545fce52b6908ea07459bcf8d33"} Sep 16 18:45:49.941397 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:49.941360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" event={"ID":"4fa27ea5-e7d8-4be1-9c00-410f85826d80","Type":"ContainerStarted","Data":"ba1765e7f4d1ebe363621fd5f05f584246776e4598bfd75c903329e64e1fcd51"} Sep 16 18:45:49.950521 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:49.950473 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:49.968566 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:49.968513 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xvmpj" podStartSLOduration=7.985308283 podStartE2EDuration="36.968491675s" totalImagesPullingTime="28.983183392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:49.966920231 +0000 UTC m=+37.766042108" watchObservedRunningTime="2026-09-16 18:45:49.968491675 +0000 UTC m=+37.767613515" Sep 16 18:45:50.062004 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:50.061920 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:50.158009 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:50.157902 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:50.260779 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:50.260705 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:50.887935 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:50.887899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:50.888129 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:50.888001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:50.888189 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:50.888141 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:50.888231 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:50.888194 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls podName:aed88e62-448f-471e-8680-79d5d962b285 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:54.888181414 +0000 UTC m=+42.687303232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls") pod "dns-default-hmp8z" (UID: "aed88e62-448f-471e-8680-79d5d962b285") : secret "dns-default-metrics-tls" not found Sep 16 18:45:50.888533 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:50.888519 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:50.888602 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:50.888533 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d455dc6cd-bp6wv: secret "image-registry-tls" not found Sep 16 18:45:50.888602 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:50.888569 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls podName:5f9e864f-101e-42ec-98bd-6b2051eb5fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:54.888558857 +0000 UTC m=+42.687680674 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls") pod "image-registry-d455dc6cd-bp6wv" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6") : secret "image-registry-tls" not found Sep 16 18:45:50.918293 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:50.918263 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:51.376472 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:51.376236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:51.494273 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:51.494235 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:52.370568 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:52.370531 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:52.543598 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:52.543493 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:54.918541 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:54.918499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:45:54.919024 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:54.918612 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:45:54.919024 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:54.918749 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:45:54.919024 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:54.918764 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d455dc6cd-bp6wv: secret "image-registry-tls" not found Sep 16 18:45:54.919024 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:54.918820 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls podName:5f9e864f-101e-42ec-98bd-6b2051eb5fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:02.918803959 +0000 UTC m=+50.717925802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls") pod "image-registry-d455dc6cd-bp6wv" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6") : secret "image-registry-tls" not found Sep 16 18:45:54.919276 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:54.919253 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:45:54.919338 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:45:54.919314 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls podName:aed88e62-448f-471e-8680-79d5d962b285 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:02.919297194 +0000 UTC m=+50.718419028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls") pod "dns-default-hmp8z" (UID: "aed88e62-448f-471e-8680-79d5d962b285") : secret "dns-default-metrics-tls" not found Sep 16 18:45:54.922254 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:54.922225 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:55.643493 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:55.643465 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:55.766438 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:55.766395 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:55.845665 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:55.845621 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:56.667439 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.667408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:56.733042 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.732976 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:56.816492 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.816459 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:56.961681 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.961654 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" event={"ID":"c555115e-10aa-47fb-8e27-6dd97a9b769c","Type":"ContainerStarted","Data":"b1353029770954f3a87dff61c5cfcd3e871a6c683493f26094667d1d55e255ca"} Sep 16 18:45:56.961827 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.961808 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:56.963348 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.963320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" event={"ID":"e0e4c7f6-27c5-4b24-b948-19547c652307","Type":"ContainerStarted","Data":"27b1bae0fa62c7027c91b729b83f8a47bf89d9670e4a77158ed04c2d64a63485"} Sep 16 18:45:56.963797 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.963777 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:45:56.964885 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.964857 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cq5vr" event={"ID":"9adce71e-476d-4ff2-9cd0-ce06e831d479","Type":"ContainerStarted","Data":"17bb3d216e76220a64c3938a5dd9173a691e53c30570229a67b0220f46eab1b1"} Sep 16 18:45:56.970126 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.970103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" event={"ID":"e430d3bb-7ea2-49f9-819b-5211dc39a851","Type":"ContainerStarted","Data":"21ec32b42d0042b0ac3d45968eca4fa0b7ecd1f53649d78c74c65f07d240420f"} Sep 16 18:45:56.970226 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.970137 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" event={"ID":"e430d3bb-7ea2-49f9-819b-5211dc39a851","Type":"ContainerStarted","Data":"b5ac1a88c7e94de9a6114e070b50ff6dee5f02d41ef616dfaad482b27f0582ad"} Sep 16 18:45:56.970226 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.970154 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" event={"ID":"e430d3bb-7ea2-49f9-819b-5211dc39a851","Type":"ContainerStarted","Data":"53bcb22008d355cb8578f9c07860604f8cf77090a4911f3221d0527df29407e0"} Sep 16 18:45:56.991986 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:56.991879 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" podStartSLOduration=31.493327664 podStartE2EDuration="39.991866085s" totalImagesPullingTime="8.498538421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:56.991496786 +0000 UTC m=+44.790618637" watchObservedRunningTime="2026-09-16 18:45:56.991866085 +0000 UTC m=+44.790987932" Sep 16 18:45:57.015765 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:57.015720 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" podStartSLOduration=31.929948221 podStartE2EDuration="40.015704911s" totalImagesPullingTime="8.08575669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:57.014759293 +0000 UTC m=+44.813881131" watchObservedRunningTime="2026-09-16 18:45:57.015704911 +0000 UTC m=+44.814826749" Sep 16 18:45:57.051977 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:57.051922 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cq5vr" podStartSLOduration=1.9657959699999998 podStartE2EDuration="10.051907966s" totalImagesPullingTime="8.086111996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:57.05060216 +0000 UTC m=+44.849723999" watchObservedRunningTime="2026-09-16 18:45:57.051907966 +0000 UTC m=+44.851029804" Sep 16 18:45:57.074381 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:57.074337 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" podStartSLOduration=31.975680939 podStartE2EDuration="40.074323244s" totalImagesPullingTime="8.098642305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:57.073941292 +0000 UTC m=+44.873063133" watchObservedRunningTime="2026-09-16 18:45:57.074323244 +0000 UTC m=+44.873445083" Sep 16 18:45:58.154666 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:58.154637 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:58.244828 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:58.244796 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:58.948261 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:58.948229 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:59.166080 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:59.166050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:45:59.250761 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:45:59.250675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:01.995978 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:01.995939 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:02.982382 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:02.982343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:46:02.982561 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:02.982408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:46:02.982561 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:02.982496 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:02.982561 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:02.982503 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:46:02.982561 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:02.982522 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d455dc6cd-bp6wv: secret "image-registry-tls" not found Sep 16 18:46:02.982692 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:02.982565 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls podName:aed88e62-448f-471e-8680-79d5d962b285 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.982547947 +0000 UTC m=+66.781669780 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls") pod "dns-default-hmp8z" (UID: "aed88e62-448f-471e-8680-79d5d962b285") : secret "dns-default-metrics-tls" not found Sep 16 18:46:02.982692 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:02.982579 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls podName:5f9e864f-101e-42ec-98bd-6b2051eb5fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.982573008 +0000 UTC m=+66.781694824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls") pod "image-registry-d455dc6cd-bp6wv" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6") : secret "image-registry-tls" not found Sep 16 18:46:05.913952 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:05.913908 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:06.015005 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:06.014951 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:06.080125 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:06.080095 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:07.244866 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:07.244829 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:07.342482 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:07.342442 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:09.730595 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:09.730564 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:09.831675 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:09.831646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:10.901245 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:10.901220 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2qjb" Sep 16 18:46:11.071605 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:11.071564 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:11.149428 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:11.149392 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:12.757594 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:12.757542 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:12.834522 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:12.834490 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:18.483121 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.483079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:46:18.483548 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.483128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:46:18.487241 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.487224 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:46:18.487348 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.487331 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:46:18.494090 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:18.494076 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:46:18.494151 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:18.494123 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs podName:3198a50c-14d2-4981-8d5d-7718876d54b1 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:22.49410964 +0000 UTC m=+130.293231470 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs") pod "network-metrics-daemon-x7nkj" (UID: "3198a50c-14d2-4981-8d5d-7718876d54b1") : secret "metrics-daemon-secret" not found Sep 16 18:46:18.497066 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.497045 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/79ae6a44-0eb1-43cc-a9ff-63bc5342a979-original-pull-secret\") pod \"global-pull-secret-syncer-sz2r4\" (UID: \"79ae6a44-0eb1-43cc-a9ff-63bc5342a979\") " pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:46:18.583642 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.583613 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:46:18.587349 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.587332 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:46:18.600374 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.600353 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:46:18.606661 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.606643 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x42tr\" (UniqueName: \"kubernetes.io/projected/db40c4ce-12ac-41c2-9b56-6a53b5616259-kube-api-access-x42tr\") pod \"network-check-target-jr5jk\" (UID: \"db40c4ce-12ac-41c2-9b56-6a53b5616259\") " pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:46:18.730225 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.730204 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sz2r4" Sep 16 18:46:18.751272 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.751251 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sqjhk\"" Sep 16 18:46:18.759008 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.758988 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:46:18.986608 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.986537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:46:18.986608 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:18.986589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:46:18.986784 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:18.986709 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:46:18.986784 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:18.986720 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:18.986784 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:18.986728 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d455dc6cd-bp6wv: secret "image-registry-tls" not found Sep 16 18:46:18.986784 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:18.986781 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls podName:5f9e864f-101e-42ec-98bd-6b2051eb5fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:50.98676225 +0000 UTC m=+98.785884067 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls") pod "image-registry-d455dc6cd-bp6wv" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6") : secret "image-registry-tls" not found Sep 16 18:46:18.986925 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:18.986835 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls podName:aed88e62-448f-471e-8680-79d5d962b285 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:50.986824492 +0000 UTC m=+98.785946308 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls") pod "dns-default-hmp8z" (UID: "aed88e62-448f-471e-8680-79d5d962b285") : secret "dns-default-metrics-tls" not found Sep 16 18:46:19.005794 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:19.005766 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sz2r4"] Sep 16 18:46:19.010554 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:46:19.008830 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ae6a44_0eb1_43cc_a9ff_63bc5342a979.slice/crio-e1711602b3c0b505e0f640473fb1b912f01a5f104f0f8aab7894175da2283756 WatchSource:0}: Error finding container e1711602b3c0b505e0f640473fb1b912f01a5f104f0f8aab7894175da2283756: Status 404 returned error can't find the container with id e1711602b3c0b505e0f640473fb1b912f01a5f104f0f8aab7894175da2283756 Sep 16 18:46:19.033485 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:19.033464 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jr5jk"] Sep 16 18:46:19.034351 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:19.034326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sz2r4" event={"ID":"79ae6a44-0eb1-43cc-a9ff-63bc5342a979","Type":"ContainerStarted","Data":"e1711602b3c0b505e0f640473fb1b912f01a5f104f0f8aab7894175da2283756"} Sep 16 18:46:19.036256 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:46:19.036235 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb40c4ce_12ac_41c2_9b56_6a53b5616259.slice/crio-9914c38f5a43e90c9e3aa1d72926db637a05e8baba447bce45be3363eba7acc1 WatchSource:0}: Error finding container 9914c38f5a43e90c9e3aa1d72926db637a05e8baba447bce45be3363eba7acc1: Status 404 returned error can't find the container with id 9914c38f5a43e90c9e3aa1d72926db637a05e8baba447bce45be3363eba7acc1 Sep 16 18:46:20.039091 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:20.039053 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jr5jk" event={"ID":"db40c4ce-12ac-41c2-9b56-6a53b5616259","Type":"ContainerStarted","Data":"9914c38f5a43e90c9e3aa1d72926db637a05e8baba447bce45be3363eba7acc1"} Sep 16 18:46:24.056410 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:24.056366 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sz2r4" event={"ID":"79ae6a44-0eb1-43cc-a9ff-63bc5342a979","Type":"ContainerStarted","Data":"3402d1a818b26191d28e0af17dcda139f3b004a7c60d68055f893e11ba2a2eb4"} Sep 16 18:46:24.057782 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:24.057757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jr5jk" event={"ID":"db40c4ce-12ac-41c2-9b56-6a53b5616259","Type":"ContainerStarted","Data":"e4881c2c7e0ffcbf288d0eb964caca3c6fec38adbbc8e5c8449c6749b12aaae9"} Sep 16 18:46:24.057901 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:24.057888 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:46:24.074394 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:24.074347 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sz2r4" podStartSLOduration=66.737286546 podStartE2EDuration="1m11.074334828s" totalImagesPullingTime="4.337048282s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:24.073144997 +0000 UTC m=+71.872266850" watchObservedRunningTime="2026-09-16 18:46:24.074334828 +0000 UTC m=+71.873456667" Sep 16 18:46:24.089828 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:24.089782 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jr5jk" podStartSLOduration=66.781829233 podStartE2EDuration="1m11.089767803s" totalImagesPullingTime="4.30793857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:24.089351855 +0000 UTC m=+71.888473720" watchObservedRunningTime="2026-09-16 18:46:24.089767803 +0000 UTC m=+71.888889662" Sep 16 18:46:27.990330 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:27.990291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:28.075752 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:28.075722 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:33.720491 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:33.720459 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:35.281785 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:35.281662 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:51.015389 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:51.015329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:46:51.015914 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:51.015413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:46:51.015914 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:51.015489 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 18:46:51.015914 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:51.015512 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d455dc6cd-bp6wv: secret "image-registry-tls" not found Sep 16 18:46:51.015914 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:51.015529 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:51.015914 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:51.015577 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls podName:5f9e864f-101e-42ec-98bd-6b2051eb5fe6 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:55.015561256 +0000 UTC m=+162.814683073 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls") pod "image-registry-d455dc6cd-bp6wv" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6") : secret "image-registry-tls" not found Sep 16 18:46:51.015914 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:46:51.015591 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls podName:aed88e62-448f-471e-8680-79d5d962b285 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:55.015585136 +0000 UTC m=+162.814706953 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls") pod "dns-default-hmp8z" (UID: "aed88e62-448f-471e-8680-79d5d962b285") : secret "dns-default-metrics-tls" not found Sep 16 18:46:55.063325 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:55.063291 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jr5jk" Sep 16 18:46:57.737740 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:57.737704 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf"] Sep 16 18:46:57.769300 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:57.769269 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf"] Sep 16 18:46:57.769448 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:57.769361 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf" Sep 16 18:46:57.772431 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:57.772413 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9wkmz\"" Sep 16 18:46:57.863328 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:57.863291 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6kg4\" (UniqueName: \"kubernetes.io/projected/127d816f-fcb2-4c92-aec9-b9c9228c7252-kube-api-access-n6kg4\") pod \"network-check-source-6f8b768455-6lmhf\" (UID: \"127d816f-fcb2-4c92-aec9-b9c9228c7252\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf" Sep 16 18:46:57.964478 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:57.964442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6kg4\" (UniqueName: \"kubernetes.io/projected/127d816f-fcb2-4c92-aec9-b9c9228c7252-kube-api-access-n6kg4\") pod \"network-check-source-6f8b768455-6lmhf\" (UID: \"127d816f-fcb2-4c92-aec9-b9c9228c7252\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf" Sep 16 18:46:57.980228 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:57.980204 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6kg4\" (UniqueName: \"kubernetes.io/projected/127d816f-fcb2-4c92-aec9-b9c9228c7252-kube-api-access-n6kg4\") pod \"network-check-source-6f8b768455-6lmhf\" (UID: \"127d816f-fcb2-4c92-aec9-b9c9228c7252\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf" Sep 16 18:46:58.032241 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:58.032143 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:58.078028 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:58.077998 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf" Sep 16 18:46:58.135081 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:58.135045 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:58.372988 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:58.372890 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf"] Sep 16 18:46:58.376105 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:46:58.376076 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod127d816f_fcb2_4c92_aec9_b9c9228c7252.slice/crio-cae2ccf51e992b922b74567e1506d90ceac3bc207718d90a471424a24c3b6260 WatchSource:0}: Error finding container cae2ccf51e992b922b74567e1506d90ceac3bc207718d90a471424a24c3b6260: Status 404 returned error can't find the container with id cae2ccf51e992b922b74567e1506d90ceac3bc207718d90a471424a24c3b6260 Sep 16 18:46:59.159286 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:59.159250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf" event={"ID":"127d816f-fcb2-4c92-aec9-b9c9228c7252","Type":"ContainerStarted","Data":"852227b6a5632ff00dce8e732f74372b945b626c5fa0b7f6fca19f16a0508bc9"} Sep 16 18:46:59.159637 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:59.159303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf" event={"ID":"127d816f-fcb2-4c92-aec9-b9c9228c7252","Type":"ContainerStarted","Data":"cae2ccf51e992b922b74567e1506d90ceac3bc207718d90a471424a24c3b6260"} Sep 16 18:46:59.178004 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:59.177938 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-6lmhf" podStartSLOduration=2.177924991 podStartE2EDuration="2.177924991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:59.177062576 +0000 UTC m=+106.976184435" watchObservedRunningTime="2026-09-16 18:46:59.177924991 +0000 UTC m=+106.977046830" Sep 16 18:46:59.848981 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:59.848925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:46:59.934071 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:46:59.934037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:02.116408 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.116372 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:02.204442 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.204397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:02.271748 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.271721 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:02.724572 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.724530 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:02.725724 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.725705 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9lbj8"] Sep 16 18:47:02.728088 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.728072 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.732255 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.732239 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 18:47:02.732354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.732241 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 18:47:02.732897 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.732882 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-46pc5\"" Sep 16 18:47:02.732949 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.732915 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 18:47:02.733016 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.732977 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 18:47:02.739782 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.739761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9lbj8"] Sep 16 18:47:02.805566 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.805533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-data-volume\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.805566 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.805571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.805762 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.805639 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.805762 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.805692 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wgp\" (UniqueName: \"kubernetes.io/projected/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-api-access-p5wgp\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.805762 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.805713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-crio-socket\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.812653 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.812623 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:02.906760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.906720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.906760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.906764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wgp\" (UniqueName: \"kubernetes.io/projected/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-api-access-p5wgp\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.907045 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.906782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-crio-socket\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.907045 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.906852 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-crio-socket\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.907045 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.906883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-data-volume\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.907045 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.906914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.907045 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:02.906937 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:02.907045 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:02.906993 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:02.907045 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:02.907024 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:03.407009899 +0000 UTC m=+111.206131715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:02.907045 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:02.907052 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:03.407038975 +0000 UTC m=+111.206160792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : configmap "kube-rbac-proxy" not found Sep 16 18:47:02.907385 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.907207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-data-volume\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:02.915864 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:02.915844 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wgp\" (UniqueName: \"kubernetes.io/projected/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-api-access-p5wgp\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:03.073050 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:03.072944 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:03.149155 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:03.149123 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:03.410252 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:03.410177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:03.410392 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:03.410262 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:03.410392 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:03.410339 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:03.410392 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:03.410339 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:03.410489 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:03.410398 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.410385407 +0000 UTC m=+112.209507225 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : configmap "kube-rbac-proxy" not found Sep 16 18:47:03.410489 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:03.410413 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:04.410408004 +0000 UTC m=+112.209529820 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:03.786322 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:03.786270 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:03.878317 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:03.878290 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:04.419032 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:04.418997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:04.419393 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:04.419046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:04.419393 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:04.419126 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:04.419393 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:04.419189 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:04.419393 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:04.419201 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:06.419184228 +0000 UTC m=+114.218306064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : configmap "kube-rbac-proxy" not found Sep 16 18:47:04.419393 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:04.419230 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:06.41921865 +0000 UTC m=+114.218340489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:05.093175 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:05.093139 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:05.161951 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:05.161910 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:05.421323 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:05.421301 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57pgm_956fd3b6-79ea-4195-8ce8-35658bd3d7de/dns-node-resolver/0.log" Sep 16 18:47:05.421749 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:05.421706 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:05.493230 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:05.493192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:06.221389 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:06.221361 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m9mfd_227b69aa-2d1b-4bf7-8603-f19fb4f2a606/node-ca/0.log" Sep 16 18:47:06.434891 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:06.434856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:06.435275 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:06.434903 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:06.435275 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:06.435000 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:06.435275 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:06.435025 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:06.435275 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:06.435068 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.435049902 +0000 UTC m=+118.234171724 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : configmap "kube-rbac-proxy" not found Sep 16 18:47:06.435275 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:06.435084 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:10.43507729 +0000 UTC m=+118.234199106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:07.023745 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:07.023710 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cq5vr_9adce71e-476d-4ff2-9cd0-ce06e831d479/serve-healthcheck-canary/0.log" Sep 16 18:47:07.246295 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:07.246264 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:09.154816 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:09.154785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:09.288087 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:09.288051 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:09.386894 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:09.386828 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:09.530307 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:09.530272 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:09.736379 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:09.736350 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:09.890937 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:09.890855 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:10.464750 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:10.464712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:10.465248 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:10.464802 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:10.465248 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:10.464867 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:10.465248 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:10.464881 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:10.465248 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:10.464914 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:18.464901392 +0000 UTC m=+126.264023223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : configmap "kube-rbac-proxy" not found Sep 16 18:47:10.465248 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:10.464958 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:18.46493936 +0000 UTC m=+126.264061189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:10.861027 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:10.860925 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:11.049370 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:11.049332 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:11.250649 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:11.250608 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:11.635193 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:11.635094 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:11.726074 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:11.726041 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:12.050593 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:12.050564 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:12.849426 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:12.849390 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:12.922602 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:12.922576 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:13.047337 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:13.047305 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:13.443347 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:13.443316 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:13.674146 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:13.674108 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:13.806410 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:13.806328 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:14.052771 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:14.052735 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:14.857120 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:14.857087 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:15.241757 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:15.241726 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:15.320364 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:15.320329 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:15.451077 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:15.451044 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:15.528795 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:15.528724 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:15.647155 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:15.647126 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:15.855459 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:15.855374 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:16.642051 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:16.642009 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:16.844509 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:16.844470 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:17.434652 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:17.434619 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:17.638840 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:17.638804 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:18.065521 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:18.065497 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:18.532872 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:18.532840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:18.533080 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:18.532902 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:18.533080 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:18.533053 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:18.533171 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:18.533137 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls podName:0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.533122787 +0000 UTC m=+142.332244605 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9lbj8" (UID: "0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:18.533339 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:18.533322 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:18.653407 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:18.653371 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:19.039222 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:19.039192 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:19.383196 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:19.383115 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:19.478650 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:19.478613 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:19.635570 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:19.635487 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:19.839067 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:19.839034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:22.561646 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:22.561590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:47:22.563816 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:22.563795 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3198a50c-14d2-4981-8d5d-7718876d54b1-metrics-certs\") pod \"network-metrics-daemon-x7nkj\" (UID: \"3198a50c-14d2-4981-8d5d-7718876d54b1\") " pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:47:22.662729 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:22.662701 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtjvb\"" Sep 16 18:47:22.667674 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:22.667656 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x7nkj" Sep 16 18:47:22.961000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:22.960954 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x7nkj"] Sep 16 18:47:22.963848 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:47:22.963820 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3198a50c_14d2_4981_8d5d_7718876d54b1.slice/crio-0d36905450f9985f24c2a463df19e5b395b6986d8ad9c4881e36fbd1adee49c4 WatchSource:0}: Error finding container 0d36905450f9985f24c2a463df19e5b395b6986d8ad9c4881e36fbd1adee49c4: Status 404 returned error can't find the container with id 0d36905450f9985f24c2a463df19e5b395b6986d8ad9c4881e36fbd1adee49c4 Sep 16 18:47:23.095295 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.095267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:23.194803 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.194761 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:23.228984 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.228886 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7nkj" event={"ID":"3198a50c-14d2-4981-8d5d-7718876d54b1","Type":"ContainerStarted","Data":"0d36905450f9985f24c2a463df19e5b395b6986d8ad9c4881e36fbd1adee49c4"} Sep 16 18:47:23.275260 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.275221 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:23.380183 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.380149 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:23.571275 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.571201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:23.651139 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.651100 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:23.764096 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.764065 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:23.874627 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.874598 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:23.998243 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:23.998206 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:24.096914 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:24.096826 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:24.197094 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:24.197067 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:24.234404 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:24.234377 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7nkj" event={"ID":"3198a50c-14d2-4981-8d5d-7718876d54b1","Type":"ContainerStarted","Data":"cc4114d2fc8674718b81c8ad6239f87c02f77845033a58763cc1d98fe9c3aafd"} Sep 16 18:47:24.234532 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:24.234421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x7nkj" event={"ID":"3198a50c-14d2-4981-8d5d-7718876d54b1","Type":"ContainerStarted","Data":"83f131daf3c2cc46c0e0aaa0317465b127907b9d9bedfb1f215614d336ef273f"} Sep 16 18:47:24.258541 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:24.258504 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x7nkj" podStartSLOduration=130.477890592 podStartE2EDuration="2m11.258490168s" totalImagesPullingTime="780.599576ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:24.257759199 +0000 UTC m=+132.056881049" watchObservedRunningTime="2026-09-16 18:47:24.258490168 +0000 UTC m=+132.057612007" Sep 16 18:47:24.788154 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:24.788113 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:24.987833 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:24.987797 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:25.150445 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:25.150364 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:25.596248 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:25.596214 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:25.663177 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:25.663149 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:29.243633 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:29.243601 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:29.387190 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:29.387152 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:29.499367 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:29.499294 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:31.983594 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:31.983538 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:32.100471 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:32.100441 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:32.630972 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:32.630942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:32.730421 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:32.730388 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:32.831882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:32.831844 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:33.180072 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:33.180040 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:33.652654 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:33.652567 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:34.552882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:34.552845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:34.555112 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:34.555088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9lbj8\" (UID: \"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0\") " pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:34.840452 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:34.840371 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-46pc5\"" Sep 16 18:47:34.848296 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:34.848274 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9lbj8" Sep 16 18:47:35.125332 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:35.125249 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9lbj8"] Sep 16 18:47:35.128271 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:47:35.128247 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4c5923_5c42_4f52_99ec_06d2d0ccc4b0.slice/crio-1c60fb3c895a62d49f2a6b8dc93a408feb9d3f7b8ac8645fbb12e49d0ea935c8 WatchSource:0}: Error finding container 1c60fb3c895a62d49f2a6b8dc93a408feb9d3f7b8ac8645fbb12e49d0ea935c8: Status 404 returned error can't find the container with id 1c60fb3c895a62d49f2a6b8dc93a408feb9d3f7b8ac8645fbb12e49d0ea935c8 Sep 16 18:47:35.271227 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:35.271189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lbj8" event={"ID":"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0","Type":"ContainerStarted","Data":"9710281ca522b3036dc58a073ab514c00fb8ba5e0cfcc818f7d64a6f397959de"} Sep 16 18:47:35.271384 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:35.271270 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lbj8" event={"ID":"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0","Type":"ContainerStarted","Data":"1c60fb3c895a62d49f2a6b8dc93a408feb9d3f7b8ac8645fbb12e49d0ea935c8"} Sep 16 18:47:35.845708 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:35.845673 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:35.939188 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:35.939157 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:36.277230 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:36.277138 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lbj8" event={"ID":"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0","Type":"ContainerStarted","Data":"6966f28f617ad69050cae4988987de063cfeae5462eff09cd556d88455602a68"} Sep 16 18:47:36.829110 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:36.829072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:36.930378 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:36.930347 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:37.028901 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:37.028862 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:37.282353 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:37.282321 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9lbj8" event={"ID":"0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0","Type":"ContainerStarted","Data":"3b8af7dd7c7b80f0b2149f9840ba66cd284459c4ab5b6e12ea93abb4fe18d60c"} Sep 16 18:47:37.324769 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:37.324719 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9lbj8" podStartSLOduration=33.678110746 podStartE2EDuration="35.324704648s" totalImagesPullingTime="1.646593902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:37.324372912 +0000 UTC m=+145.123494750" watchObservedRunningTime="2026-09-16 18:47:37.324704648 +0000 UTC m=+145.123826512" Sep 16 18:47:39.198450 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:39.198422 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:39.309325 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:39.309296 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:39.408662 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:39.408628 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:41.029960 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.029923 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:41.178649 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.178618 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:41.280383 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.280306 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:41.497841 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.497813 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:41.603717 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.603638 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nkhsn"] Sep 16 18:47:41.606757 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.606742 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.616801 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:41.616776 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"metrics-client-ca\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" Sep 16 18:47:41.616911 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:41.616817 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"node-exporter-accelerators-collector-config\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" Sep 16 18:47:41.616911 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:41.616823 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-kube-rbac-proxy-config\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" Sep 16 18:47:41.622684 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:41.622665 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Sep 16 18:47:41.622741 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:41.622714 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-dockercfg-vbsp9\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vbsp9\"" type="*v1.Secret" Sep 16 18:47:41.623069 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:41.623048 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"node-exporter-nkhsn\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" podUID="a551dd06-0827-481f-9229-41f146c34ac3" pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.623118 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:41.623083 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Sep 16 18:47:41.623118 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:41.623107 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-140-230.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-140-230.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" Sep 16 18:47:41.666571 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.666532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:41.706824 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.706791 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-sys\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.707009 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.706850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.707009 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.706920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-textfile\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.707009 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.706953 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-metrics-client-ca\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.707009 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.706999 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.707187 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.707052 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-tls\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.707187 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.707092 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-288rp\" (UniqueName: \"kubernetes.io/projected/a551dd06-0827-481f-9229-41f146c34ac3-kube-api-access-288rp\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.707187 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.707112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-root\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.707187 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.707131 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-wtmp\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-textfile\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-metrics-client-ca\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807908 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807908 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807713 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-tls\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807908 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288rp\" (UniqueName: \"kubernetes.io/projected/a551dd06-0827-481f-9229-41f146c34ac3-kube-api-access-288rp\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807908 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-root\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807908 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-wtmp\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807908 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807828 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-sys\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807908 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807858 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-root\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.807908 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807892 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.808198 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-sys\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.808198 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.807931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-wtmp\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.808198 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.808066 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-textfile\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:41.850087 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.850063 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:41.944671 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:41.944636 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:42.045659 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.045625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:42.157865 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.157831 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:42.314501 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.314412 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:42.414786 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.414756 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:42.589900 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.589820 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 18:47:42.600687 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.600665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-tls\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:42.609348 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.609324 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:42.687790 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.687762 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 18:47:42.748815 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.748784 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 18:47:42.758576 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.758551 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-metrics-client-ca\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:42.761956 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.761935 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:42.808112 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:42.808070 2559 configmap.go:193] Couldn't get configMap openshift-monitoring/node-exporter-accelerators-collector-config: failed to sync configmap cache: timed out waiting for the condition Sep 16 18:47:42.808279 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:42.808158 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-accelerators-collector-config podName:a551dd06-0827-481f-9229-41f146c34ac3 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.308142272 +0000 UTC m=+151.107264089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-accelerators-collector-config" (UniqueName: "kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-accelerators-collector-config") pod "node-exporter-nkhsn" (UID: "a551dd06-0827-481f-9229-41f146c34ac3") : failed to sync configmap cache: timed out waiting for the condition Sep 16 18:47:42.808279 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:42.808080 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-kube-rbac-proxy-config: failed to sync secret cache: timed out waiting for the condition Sep 16 18:47:42.808279 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:42.808250 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-kube-rbac-proxy-config podName:a551dd06-0827-481f-9229-41f146c34ac3 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.308231821 +0000 UTC m=+151.107353652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-kube-rbac-proxy-config" (UniqueName: "kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-kube-rbac-proxy-config") pod "node-exporter-nkhsn" (UID: "a551dd06-0827-481f-9229-41f146c34ac3") : failed to sync secret cache: timed out waiting for the condition Sep 16 18:47:42.821369 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:42.821348 2559 projected.go:291] Couldn't get configMap openshift-monitoring/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition Sep 16 18:47:42.821458 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:42.821377 2559 projected.go:196] Error preparing data for projected volume kube-api-access-288rp for pod openshift-monitoring/node-exporter-nkhsn: failed to sync configmap cache: timed out waiting for the condition Sep 16 18:47:42.821458 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:42.821420 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a551dd06-0827-481f-9229-41f146c34ac3-kube-api-access-288rp podName:a551dd06-0827-481f-9229-41f146c34ac3 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:43.321408279 +0000 UTC m=+151.120530100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-288rp" (UniqueName: "kubernetes.io/projected/a551dd06-0827-481f-9229-41f146c34ac3-kube-api-access-288rp") pod "node-exporter-nkhsn" (UID: "a551dd06-0827-481f-9229-41f146c34ac3") : failed to sync configmap cache: timed out waiting for the condition Sep 16 18:47:42.881052 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.880953 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vbsp9\"" Sep 16 18:47:42.910552 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:42.910529 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 18:47:43.086723 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.086687 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 18:47:43.204928 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.204896 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 18:47:43.318986 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.318933 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:43.319148 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.318995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:43.319449 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.319429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-accelerators-collector-config\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:43.321136 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.321118 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a551dd06-0827-481f-9229-41f146c34ac3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:43.419604 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.419560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-288rp\" (UniqueName: \"kubernetes.io/projected/a551dd06-0827-481f-9229-41f146c34ac3-kube-api-access-288rp\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:43.421853 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.421825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-288rp\" (UniqueName: \"kubernetes.io/projected/a551dd06-0827-481f-9229-41f146c34ac3-kube-api-access-288rp\") pod \"node-exporter-nkhsn\" (UID: \"a551dd06-0827-481f-9229-41f146c34ac3\") " pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:43.715339 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:43.715306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nkhsn" Sep 16 18:47:43.726455 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:47:43.726424 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda551dd06_0827_481f_9229_41f146c34ac3.slice/crio-5703aa17f2ec561e611742e032e87fadd5787bd9f8ee76c95416a2fcd55fb44c WatchSource:0}: Error finding container 5703aa17f2ec561e611742e032e87fadd5787bd9f8ee76c95416a2fcd55fb44c: Status 404 returned error can't find the container with id 5703aa17f2ec561e611742e032e87fadd5787bd9f8ee76c95416a2fcd55fb44c Sep 16 18:47:44.237201 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:44.237166 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:44.307126 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:44.307094 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkhsn" event={"ID":"a551dd06-0827-481f-9229-41f146c34ac3","Type":"ContainerStarted","Data":"5703aa17f2ec561e611742e032e87fadd5787bd9f8ee76c95416a2fcd55fb44c"} Sep 16 18:47:44.345443 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:44.345400 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:44.436078 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:44.436035 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:44.696788 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:44.696763 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:44.985499 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:44.985413 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:45.311753 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:45.311673 2559 generic.go:356] "Generic (PLEG): container finished" podID="a551dd06-0827-481f-9229-41f146c34ac3" containerID="0941ab055dc6d023a782cd7a9c8057e79c920f297897429de46a392e26a0a460" exitCode=0 Sep 16 18:47:45.311753 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:45.311736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkhsn" event={"ID":"a551dd06-0827-481f-9229-41f146c34ac3","Type":"ContainerDied","Data":"0941ab055dc6d023a782cd7a9c8057e79c920f297897429de46a392e26a0a460"} Sep 16 18:47:46.075174 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:46.075131 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:46.166090 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:46.166050 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:46.262446 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:46.262412 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:46.317184 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:46.317156 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkhsn" event={"ID":"a551dd06-0827-481f-9229-41f146c34ac3","Type":"ContainerStarted","Data":"00f5dadb32a91a767d29150b58c5f3602584f89e63c36c3fb52fcb9a86e82146"} Sep 16 18:47:46.317553 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:46.317203 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nkhsn" event={"ID":"a551dd06-0827-481f-9229-41f146c34ac3","Type":"ContainerStarted","Data":"acc119d9a22bd90129e603f84881622d3a9c4fa610af42dcb03f11cf23f37f74"} Sep 16 18:47:46.410185 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:46.410087 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nkhsn" podStartSLOduration=4.410230022 podStartE2EDuration="5.410069885s" totalImagesPullingTime="938.331999ms" totalInitContainerRuntime="61.507864ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:46.409194783 +0000 UTC m=+154.208316641" watchObservedRunningTime="2026-09-16 18:47:46.410069885 +0000 UTC m=+154.209191723" Sep 16 18:47:46.499014 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:46.498981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:47.855250 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:47.855216 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:50.091844 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:50.091794 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" podUID="5f9e864f-101e-42ec-98bd-6b2051eb5fe6" Sep 16 18:47:50.166682 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:47:50.166631 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hmp8z" podUID="aed88e62-448f-471e-8680-79d5d962b285" Sep 16 18:47:50.329443 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:50.329418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:47:50.329636 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:50.329619 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmp8z" Sep 16 18:47:51.014466 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:51.014437 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:51.119823 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:51.119795 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:47:55.115080 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.115040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:47:55.115694 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.115102 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:47:55.117421 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.117387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/aed88e62-448f-471e-8680-79d5d962b285-metrics-tls\") pod \"dns-default-hmp8z\" (UID: \"aed88e62-448f-471e-8680-79d5d962b285\") " pod="openshift-dns/dns-default-hmp8z" Sep 16 18:47:55.117522 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.117430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"image-registry-d455dc6cd-bp6wv\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:47:55.136357 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.136331 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k8559\"" Sep 16 18:47:55.136480 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.136338 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-htp7b\"" Sep 16 18:47:55.141131 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.141111 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:47:55.141197 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.141141 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmp8z" Sep 16 18:47:55.435640 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.435606 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmp8z"] Sep 16 18:47:55.442924 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:47:55.442897 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed88e62_448f_471e_8680_79d5d962b285.slice/crio-1299991acf6d21723696a1fdf1d1393564b36147570978049f27945632b7f3df WatchSource:0}: Error finding container 1299991acf6d21723696a1fdf1d1393564b36147570978049f27945632b7f3df: Status 404 returned error can't find the container with id 1299991acf6d21723696a1fdf1d1393564b36147570978049f27945632b7f3df Sep 16 18:47:55.443457 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:55.443431 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d455dc6cd-bp6wv"] Sep 16 18:47:55.446763 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:47:55.446741 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9e864f_101e_42ec_98bd_6b2051eb5fe6.slice/crio-02645df41cf07cf612769a6a8908aee1a3f08038c7bada69f417493434da151a WatchSource:0}: Error finding container 02645df41cf07cf612769a6a8908aee1a3f08038c7bada69f417493434da151a: Status 404 returned error can't find the container with id 02645df41cf07cf612769a6a8908aee1a3f08038c7bada69f417493434da151a Sep 16 18:47:56.350843 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.350804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmp8z" event={"ID":"aed88e62-448f-471e-8680-79d5d962b285","Type":"ContainerStarted","Data":"1299991acf6d21723696a1fdf1d1393564b36147570978049f27945632b7f3df"} Sep 16 18:47:56.352807 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.352745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" event={"ID":"5f9e864f-101e-42ec-98bd-6b2051eb5fe6","Type":"ContainerStarted","Data":"f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e"} Sep 16 18:47:56.352807 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.352789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" event={"ID":"5f9e864f-101e-42ec-98bd-6b2051eb5fe6","Type":"ContainerStarted","Data":"02645df41cf07cf612769a6a8908aee1a3f08038c7bada69f417493434da151a"} Sep 16 18:47:56.352979 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.352871 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:47:56.354476 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.354452 2559 generic.go:356] "Generic (PLEG): container finished" podID="e0e4c7f6-27c5-4b24-b948-19547c652307" containerID="27b1bae0fa62c7027c91b729b83f8a47bf89d9670e4a77158ed04c2d64a63485" exitCode=255 Sep 16 18:47:56.354588 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.354529 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" event={"ID":"e0e4c7f6-27c5-4b24-b948-19547c652307","Type":"ContainerDied","Data":"27b1bae0fa62c7027c91b729b83f8a47bf89d9670e4a77158ed04c2d64a63485"} Sep 16 18:47:56.354820 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.354798 2559 scope.go:117] "RemoveContainer" containerID="27b1bae0fa62c7027c91b729b83f8a47bf89d9670e4a77158ed04c2d64a63485" Sep 16 18:47:56.420493 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.420440 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" podStartSLOduration=162.420416239 podStartE2EDuration="2m42.420416239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:56.38617474 +0000 UTC m=+164.185296599" watchObservedRunningTime="2026-09-16 18:47:56.420416239 +0000 UTC m=+164.219538080" Sep 16 18:47:56.962829 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:56.962786 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" podUID="c555115e-10aa-47fb-8e27-6dd97a9b769c" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused" Sep 16 18:47:57.359540 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.359504 2559 generic.go:356] "Generic (PLEG): container finished" podID="c555115e-10aa-47fb-8e27-6dd97a9b769c" containerID="b1353029770954f3a87dff61c5cfcd3e871a6c683493f26094667d1d55e255ca" exitCode=1 Sep 16 18:47:57.360025 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.359578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" event={"ID":"c555115e-10aa-47fb-8e27-6dd97a9b769c","Type":"ContainerDied","Data":"b1353029770954f3a87dff61c5cfcd3e871a6c683493f26094667d1d55e255ca"} Sep 16 18:47:57.360025 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.359924 2559 scope.go:117] "RemoveContainer" containerID="b1353029770954f3a87dff61c5cfcd3e871a6c683493f26094667d1d55e255ca" Sep 16 18:47:57.361749 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.361728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-69d9d7484d-bt9zp" event={"ID":"e0e4c7f6-27c5-4b24-b948-19547c652307","Type":"ContainerStarted","Data":"715d37523d8402ed63be7674b23eb16f788fc9e780c51020b433b8e2f244f04d"} Sep 16 18:47:57.363744 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.363723 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmp8z" event={"ID":"aed88e62-448f-471e-8680-79d5d962b285","Type":"ContainerStarted","Data":"b646fbf5108ec5b816fdf00b93a2063dc5cdf43d9ffd9450e2643e51ae9792ee"} Sep 16 18:47:57.363843 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.363767 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmp8z" event={"ID":"aed88e62-448f-471e-8680-79d5d962b285","Type":"ContainerStarted","Data":"e7c3f167479dc112583ad72c783d7355af322db512751ed22ea968294e21292a"} Sep 16 18:47:57.363955 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.363940 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hmp8z" Sep 16 18:47:57.422884 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.422844 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hmp8z" podStartSLOduration=128.982464857 podStartE2EDuration="2m10.42282873s" totalImagesPullingTime="1.440363873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:57.422106276 +0000 UTC m=+165.221228126" watchObservedRunningTime="2026-09-16 18:47:57.42282873 +0000 UTC m=+165.221950624" Sep 16 18:47:57.440104 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:57.440082 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:47:58.368939 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:58.368896 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" event={"ID":"c555115e-10aa-47fb-8e27-6dd97a9b769c","Type":"ContainerStarted","Data":"b95dad68ef31c8d3f001c26a2106125186ab8f417fdbaf1f44b75151adde03c8"} Sep 16 18:47:58.369431 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:58.369204 2559 status_manager.go:444] "Container readiness changed for unknown container" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" containerID="cri-o://b1353029770954f3a87dff61c5cfcd3e871a6c683493f26094667d1d55e255ca" Sep 16 18:47:58.369431 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:58.369223 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:47:59.372371 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:59.372340 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:47:59.373317 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:47:59.373296 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-64cd77df94-hhckp" Sep 16 18:48:00.663235 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:00.663202 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:00.737004 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:00.736949 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:02.199671 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:02.199637 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:02.281318 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:02.281284 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:07.371889 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:07.371811 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hmp8z" Sep 16 18:48:10.109446 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:10.109411 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:10.204778 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:10.204742 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:14.945866 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:14.945832 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:15.145772 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:15.145739 2559 patch_prober.go:28] interesting pod/image-registry-d455dc6cd-bp6wv container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 16 18:48:15.145924 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:15.145787 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" podUID="5f9e864f-101e-42ec-98bd-6b2051eb5fe6" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:48:15.156100 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:15.156072 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:15.257372 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:15.257283 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:16.375673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:16.375636 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:16.474610 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:16.474566 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:16.576442 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:16.576408 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:16.647251 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:16.647169 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:17.191441 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:17.191407 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:17.289727 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:17.289697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:17.368513 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:17.368483 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:48:17.623112 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:17.623034 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:17.721652 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:17.721624 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:18.170182 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:18.170146 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:18.279244 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:18.279209 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:19.498411 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:19.498376 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:19.583389 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:19.583354 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:19.739843 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:19.739806 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d455dc6cd-bp6wv"] Sep 16 18:48:19.860161 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:19.860068 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:19.959304 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:19.959274 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:20.040195 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:20.040167 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:28.684130 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:28.683933 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:28.798178 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:28.798141 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:29.184410 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:29.184376 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:29.314677 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:29.314649 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:31.455885 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:31.455855 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:31.574733 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:31.574697 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:36.659322 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:36.659285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:36.732299 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:36.732267 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:37.202828 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:37.202786 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:37.445579 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:37.445545 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" podUID="e430d3bb-7ea2-49f9-819b-5211dc39a851" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 18:48:40.439105 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:40.439070 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:40.504259 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:40.504228 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:44.148850 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:44.148813 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:44.246027 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:44.245956 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:44.759478 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:44.759437 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" podUID="5f9e864f-101e-42ec-98bd-6b2051eb5fe6" containerName="registry" containerID="cri-o://f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e" gracePeriod=30 Sep 16 18:48:45.993994 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:45.993953 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:48:46.077149 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077119 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-certificates\") pod \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " Sep 16 18:48:46.077283 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077160 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-trusted-ca\") pod \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " Sep 16 18:48:46.077283 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-ca-trust-extracted\") pod \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " Sep 16 18:48:46.077425 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077393 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-bound-sa-token\") pod \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " Sep 16 18:48:46.077551 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077466 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-installation-pull-secrets\") pod \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " Sep 16 18:48:46.077551 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077495 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") pod \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " Sep 16 18:48:46.077551 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077542 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-image-registry-private-configuration\") pod \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " Sep 16 18:48:46.077720 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077562 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5f9e864f-101e-42ec-98bd-6b2051eb5fe6" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:48:46.077720 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077568 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztsnc\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-kube-api-access-ztsnc\") pod \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\" (UID: \"5f9e864f-101e-42ec-98bd-6b2051eb5fe6\") " Sep 16 18:48:46.077720 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077593 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5f9e864f-101e-42ec-98bd-6b2051eb5fe6" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:48:46.077884 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077803 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-certificates\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:48:46.077884 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.077823 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-trusted-ca\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:48:46.079852 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.079819 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-kube-api-access-ztsnc" (OuterVolumeSpecName: "kube-api-access-ztsnc") pod "5f9e864f-101e-42ec-98bd-6b2051eb5fe6" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6"). InnerVolumeSpecName "kube-api-access-ztsnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:46.079852 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.079838 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5f9e864f-101e-42ec-98bd-6b2051eb5fe6" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:48:46.080031 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.079934 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5f9e864f-101e-42ec-98bd-6b2051eb5fe6" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:46.080094 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.080025 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5f9e864f-101e-42ec-98bd-6b2051eb5fe6" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:48:46.080145 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.080106 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5f9e864f-101e-42ec-98bd-6b2051eb5fe6" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:48:46.086102 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.086079 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5f9e864f-101e-42ec-98bd-6b2051eb5fe6" (UID: "5f9e864f-101e-42ec-98bd-6b2051eb5fe6"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:48:46.178506 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.178462 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-image-registry-private-configuration\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:48:46.178506 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.178503 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztsnc\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-kube-api-access-ztsnc\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:48:46.178506 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.178513 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-ca-trust-extracted\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:48:46.178506 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.178521 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-bound-sa-token\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:48:46.178735 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.178530 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-installation-pull-secrets\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:48:46.178735 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.178538 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f9e864f-101e-42ec-98bd-6b2051eb5fe6-registry-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:48:46.537792 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.537758 2559 generic.go:356] "Generic (PLEG): container finished" podID="5f9e864f-101e-42ec-98bd-6b2051eb5fe6" containerID="f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e" exitCode=0 Sep 16 18:48:46.537950 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.537804 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" Sep 16 18:48:46.537950 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.537810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" event={"ID":"5f9e864f-101e-42ec-98bd-6b2051eb5fe6","Type":"ContainerDied","Data":"f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e"} Sep 16 18:48:46.537950 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.537849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d455dc6cd-bp6wv" event={"ID":"5f9e864f-101e-42ec-98bd-6b2051eb5fe6","Type":"ContainerDied","Data":"02645df41cf07cf612769a6a8908aee1a3f08038c7bada69f417493434da151a"} Sep 16 18:48:46.537950 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.537868 2559 scope.go:117] "RemoveContainer" containerID="f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e" Sep 16 18:48:46.546099 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.546085 2559 scope.go:117] "RemoveContainer" containerID="f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e" Sep 16 18:48:46.546342 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:48:46.546326 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e\": container with ID starting with f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e not found: ID does not exist" containerID="f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e" Sep 16 18:48:46.546392 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.546348 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e"} err="failed to get container status \"f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e\": rpc error: code = NotFound desc = could not find container \"f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e\": container with ID starting with f3c4b108b5ffbc07619c3186f201156ecfa7cf1b1b3d01ddf6746b5fd40bab4e not found: ID does not exist" Sep 16 18:48:46.577434 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.577414 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d455dc6cd-bp6wv"] Sep 16 18:48:46.587030 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.587010 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d455dc6cd-bp6wv"] Sep 16 18:48:46.720490 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:46.720463 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f9e864f-101e-42ec-98bd-6b2051eb5fe6" path="/var/lib/kubelet/pods/5f9e864f-101e-42ec-98bd-6b2051eb5fe6/volumes" Sep 16 18:48:47.445685 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:47.445649 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" podUID="e430d3bb-7ea2-49f9-819b-5211dc39a851" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 18:48:57.446008 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:57.445954 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" podUID="e430d3bb-7ea2-49f9-819b-5211dc39a851" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 16 18:48:57.446350 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:57.446032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" Sep 16 18:48:57.446457 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:57.446440 2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"21ec32b42d0042b0ac3d45968eca4fa0b7ecd1f53649d78c74c65f07d240420f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 16 18:48:57.446494 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:57.446477 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" podUID="e430d3bb-7ea2-49f9-819b-5211dc39a851" containerName="service-proxy" containerID="cri-o://21ec32b42d0042b0ac3d45968eca4fa0b7ecd1f53649d78c74c65f07d240420f" gracePeriod=30 Sep 16 18:48:57.564163 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:57.564142 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:57.577288 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:57.577262 2559 generic.go:356] "Generic (PLEG): container finished" podID="e430d3bb-7ea2-49f9-819b-5211dc39a851" containerID="21ec32b42d0042b0ac3d45968eca4fa0b7ecd1f53649d78c74c65f07d240420f" exitCode=2 Sep 16 18:48:57.577384 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:57.577340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" event={"ID":"e430d3bb-7ea2-49f9-819b-5211dc39a851","Type":"ContainerDied","Data":"21ec32b42d0042b0ac3d45968eca4fa0b7ecd1f53649d78c74c65f07d240420f"} Sep 16 18:48:57.677804 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:57.677764 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:58.583756 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:58.583722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d468b69db-lw6t2" event={"ID":"e430d3bb-7ea2-49f9-819b-5211dc39a851","Type":"ContainerStarted","Data":"fd26d55110b51396903502ee1b6f2c954f9b84e5d6f8d41317db1bb24f5f8a58"} Sep 16 18:48:59.188243 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:59.188203 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:48:59.508312 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:48:59.508231 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:00.736392 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:00.736357 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:00.876199 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:00.876171 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:01.783657 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:01.783618 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:09.601018 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:09.600974 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:09.697598 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:09.697561 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:14.149444 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:14.149404 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:14.247826 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:14.247795 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:20.628888 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:20.628851 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:20.729798 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:20.729766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:20.797840 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:20.797812 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:52.478257 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:52.478223 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:49:52.580698 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:49:52.580670 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:03.225079 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:03.225037 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:12.590271 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:12.590242 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 18:50:12.594285 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:12.594263 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 18:50:12.595180 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:12.595156 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 18:50:15.993342 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:15.993302 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:16.065687 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:16.065649 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:22.725799 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:22.725764 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:22.811092 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:22.811055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:27.974851 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:27.974823 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:28.053940 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:28.053900 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:33.907091 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:33.907053 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:34.001193 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:34.001165 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:34.102693 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:34.102665 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:34.884522 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:34.884490 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:35.000374 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:35.000339 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:50:35.072386 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:50:35.072349 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:08.926958 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:08.926882 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:23.639141 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:23.639107 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:23.736172 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:23.736131 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:29.712827 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:29.712788 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:29.814560 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:29.814532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:29.914518 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:29.914485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:52.367752 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:52.367712 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:53.794441 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:53.794403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:58.473522 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:58.473483 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:58.886028 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:58.885929 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:59.488798 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.488764 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:59.795664 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.795579 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:51:59.919318 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.919289 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb"] Sep 16 18:51:59.919513 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.919502 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f9e864f-101e-42ec-98bd-6b2051eb5fe6" containerName="registry" Sep 16 18:51:59.919555 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.919517 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f9e864f-101e-42ec-98bd-6b2051eb5fe6" containerName="registry" Sep 16 18:51:59.919588 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.919578 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f9e864f-101e-42ec-98bd-6b2051eb5fe6" containerName="registry" Sep 16 18:51:59.922140 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.922125 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:51:59.933427 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.933408 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Sep 16 18:51:59.933516 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.933431 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Sep 16 18:51:59.933516 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.933409 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\"" Sep 16 18:51:59.934858 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.934839 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Sep 16 18:51:59.935011 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.934845 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-ztfjw\"" Sep 16 18:51:59.935011 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.934915 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Sep 16 18:51:59.959862 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:51:59.959835 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb"] Sep 16 18:52:00.037201 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.037170 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:00.051583 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.051505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c213b952-e51f-4549-a55c-99c9d17969b6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.051583 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.051542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c213b952-e51f-4549-a55c-99c9d17969b6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.051754 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.051582 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw5tr\" (UniqueName: \"kubernetes.io/projected/c213b952-e51f-4549-a55c-99c9d17969b6-kube-api-access-nw5tr\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.142684 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.142646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:00.152642 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.152620 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c213b952-e51f-4549-a55c-99c9d17969b6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.152707 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.152690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw5tr\" (UniqueName: \"kubernetes.io/projected/c213b952-e51f-4549-a55c-99c9d17969b6-kube-api-access-nw5tr\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.152757 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.152743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c213b952-e51f-4549-a55c-99c9d17969b6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.153110 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.153094 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/c213b952-e51f-4549-a55c-99c9d17969b6-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.154995 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.154976 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/c213b952-e51f-4549-a55c-99c9d17969b6-certificates\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.177094 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.177068 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw5tr\" (UniqueName: \"kubernetes.io/projected/c213b952-e51f-4549-a55c-99c9d17969b6-kube-api-access-nw5tr\") pod \"keda-metrics-apiserver-77857b7b98-dnbdb\" (UID: \"c213b952-e51f-4549-a55c-99c9d17969b6\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.231019 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.230990 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:00.262742 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.262709 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:00.358896 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.358811 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:00.538335 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.538304 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb"] Sep 16 18:52:00.541729 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:52:00.541696 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc213b952_e51f_4549_a55c_99c9d17969b6.slice/crio-a296913fb56fcd7382a4fdc9185dd4f7473ab973a68bcf5cf6fc717dfc0065c6 WatchSource:0}: Error finding container a296913fb56fcd7382a4fdc9185dd4f7473ab973a68bcf5cf6fc717dfc0065c6: Status 404 returned error can't find the container with id a296913fb56fcd7382a4fdc9185dd4f7473ab973a68bcf5cf6fc717dfc0065c6 Sep 16 18:52:00.543040 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.543025 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:52:00.802692 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:00.802664 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:01.165975 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:01.165943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" event={"ID":"c213b952-e51f-4549-a55c-99c9d17969b6","Type":"ContainerStarted","Data":"a296913fb56fcd7382a4fdc9185dd4f7473ab973a68bcf5cf6fc717dfc0065c6"} Sep 16 18:52:01.276333 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:01.276296 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:01.577469 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:01.577374 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:01.686018 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:01.685984 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:04.183789 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:04.183754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" event={"ID":"c213b952-e51f-4549-a55c-99c9d17969b6","Type":"ContainerStarted","Data":"cf1cab65576a1924f1226387f501b3276646c9d4800baed262f7e5da88bd6a55"} Sep 16 18:52:04.184206 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:04.183844 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:15.191743 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:15.191712 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" Sep 16 18:52:15.217083 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:15.217027 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-dnbdb" podStartSLOduration=12.958519678 podStartE2EDuration="16.217010708s" totalImagesPullingTime="3.25849103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:52:04.225168364 +0000 UTC m=+412.024290226" watchObservedRunningTime="2026-09-16 18:52:15.217010708 +0000 UTC m=+423.016132548" Sep 16 18:52:51.302842 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:51.302810 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:52:51.401689 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:52:51.401659 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:07.763810 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:07.763775 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:07.830562 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:07.830533 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:16.386846 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:16.386814 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:16.458419 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:16.458390 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:18.190020 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:18.189981 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:18.258789 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:18.258760 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.055877 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.055821 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.137722 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.137694 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-cxgfb"] Sep 16 18:53:21.138507 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.138480 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.140672 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.140657 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:21.143830 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.143812 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-fjbxk\"" Sep 16 18:53:21.144833 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.144818 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 18:53:21.145212 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.145197 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Sep 16 18:53:21.145335 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.145321 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 18:53:21.157150 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.157126 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-cxgfb"] Sep 16 18:53:21.218053 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.218025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ecd7b12-c828-46e2-92a8-d087ac774ba5-cert\") pod \"kserve-controller-manager-75c4c6c467-cxgfb\" (UID: \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\") " pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:21.218053 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.218056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hm9\" (UniqueName: \"kubernetes.io/projected/3ecd7b12-c828-46e2-92a8-d087ac774ba5-kube-api-access-97hm9\") pod \"kserve-controller-manager-75c4c6c467-cxgfb\" (UID: \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\") " pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:21.220258 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.220227 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.284016 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.283957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.318799 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.318731 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ecd7b12-c828-46e2-92a8-d087ac774ba5-cert\") pod \"kserve-controller-manager-75c4c6c467-cxgfb\" (UID: \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\") " pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:21.318799 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.318762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hm9\" (UniqueName: \"kubernetes.io/projected/3ecd7b12-c828-46e2-92a8-d087ac774ba5-kube-api-access-97hm9\") pod \"kserve-controller-manager-75c4c6c467-cxgfb\" (UID: \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\") " pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:21.321084 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.321064 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ecd7b12-c828-46e2-92a8-d087ac774ba5-cert\") pod \"kserve-controller-manager-75c4c6c467-cxgfb\" (UID: \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\") " pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:21.327652 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.327630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hm9\" (UniqueName: \"kubernetes.io/projected/3ecd7b12-c828-46e2-92a8-d087ac774ba5-kube-api-access-97hm9\") pod \"kserve-controller-manager-75c4c6c467-cxgfb\" (UID: \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\") " pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:21.387381 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.387347 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.449696 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.449670 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:21.497746 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.497713 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.719870 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.719839 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-cxgfb"] Sep 16 18:53:21.723177 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:53:21.723148 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecd7b12_c828_46e2_92a8_d087ac774ba5.slice/crio-5f8db5da66e942f2cae7a29117cb6902842da0bc35a6fd5a987731068195e844 WatchSource:0}: Error finding container 5f8db5da66e942f2cae7a29117cb6902842da0bc35a6fd5a987731068195e844: Status 404 returned error can't find the container with id 5f8db5da66e942f2cae7a29117cb6902842da0bc35a6fd5a987731068195e844 Sep 16 18:53:21.793769 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.793716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.891906 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.891857 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:21.952303 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:21.952268 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:53:22.457927 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:22.457889 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" event={"ID":"3ecd7b12-c828-46e2-92a8-d087ac774ba5","Type":"ContainerStarted","Data":"5f8db5da66e942f2cae7a29117cb6902842da0bc35a6fd5a987731068195e844"} Sep 16 18:53:24.466924 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:24.466892 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" event={"ID":"3ecd7b12-c828-46e2-92a8-d087ac774ba5","Type":"ContainerStarted","Data":"cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a"} Sep 16 18:53:24.467308 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:24.467030 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:53:24.487325 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:24.487283 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" podStartSLOduration=1.159096884 podStartE2EDuration="3.487268347s" totalImagesPullingTime="2.328171463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:24.486292736 +0000 UTC m=+492.285414576" watchObservedRunningTime="2026-09-16 18:53:24.487268347 +0000 UTC m=+492.286390198" Sep 16 18:53:55.475728 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:53:55.475691 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:54:05.506247 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.506173 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-cxgfb"] Sep 16 18:54:05.506599 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.506376 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" podUID="3ecd7b12-c828-46e2-92a8-d087ac774ba5" containerName="manager" containerID="cri-o://cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a" gracePeriod=10 Sep 16 18:54:05.535269 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.535243 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-qx55p"] Sep 16 18:54:05.537205 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.537191 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:05.548910 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.548866 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-qx55p"] Sep 16 18:54:05.631802 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.631777 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nq7d\" (UniqueName: \"kubernetes.io/projected/ee1560d2-5746-4918-bcdb-8e95be0526ab-kube-api-access-7nq7d\") pod \"kserve-controller-manager-75c4c6c467-qx55p\" (UID: \"ee1560d2-5746-4918-bcdb-8e95be0526ab\") " pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:05.631918 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.631831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee1560d2-5746-4918-bcdb-8e95be0526ab-cert\") pod \"kserve-controller-manager-75c4c6c467-qx55p\" (UID: \"ee1560d2-5746-4918-bcdb-8e95be0526ab\") " pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:05.733004 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.732960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nq7d\" (UniqueName: \"kubernetes.io/projected/ee1560d2-5746-4918-bcdb-8e95be0526ab-kube-api-access-7nq7d\") pod \"kserve-controller-manager-75c4c6c467-qx55p\" (UID: \"ee1560d2-5746-4918-bcdb-8e95be0526ab\") " pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:05.733110 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.733045 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee1560d2-5746-4918-bcdb-8e95be0526ab-cert\") pod \"kserve-controller-manager-75c4c6c467-qx55p\" (UID: \"ee1560d2-5746-4918-bcdb-8e95be0526ab\") " pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:05.735115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.735095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ee1560d2-5746-4918-bcdb-8e95be0526ab-cert\") pod \"kserve-controller-manager-75c4c6c467-qx55p\" (UID: \"ee1560d2-5746-4918-bcdb-8e95be0526ab\") " pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:05.741464 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.741440 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:54:05.742000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.741982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nq7d\" (UniqueName: \"kubernetes.io/projected/ee1560d2-5746-4918-bcdb-8e95be0526ab-kube-api-access-7nq7d\") pod \"kserve-controller-manager-75c4c6c467-qx55p\" (UID: \"ee1560d2-5746-4918-bcdb-8e95be0526ab\") " pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:05.833857 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.833780 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ecd7b12-c828-46e2-92a8-d087ac774ba5-cert\") pod \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\" (UID: \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\") " Sep 16 18:54:05.833857 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.833848 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97hm9\" (UniqueName: \"kubernetes.io/projected/3ecd7b12-c828-46e2-92a8-d087ac774ba5-kube-api-access-97hm9\") pod \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\" (UID: \"3ecd7b12-c828-46e2-92a8-d087ac774ba5\") " Sep 16 18:54:05.835889 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.835855 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ecd7b12-c828-46e2-92a8-d087ac774ba5-cert" (OuterVolumeSpecName: "cert") pod "3ecd7b12-c828-46e2-92a8-d087ac774ba5" (UID: "3ecd7b12-c828-46e2-92a8-d087ac774ba5"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:54:05.836020 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.835899 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ecd7b12-c828-46e2-92a8-d087ac774ba5-kube-api-access-97hm9" (OuterVolumeSpecName: "kube-api-access-97hm9") pod "3ecd7b12-c828-46e2-92a8-d087ac774ba5" (UID: "3ecd7b12-c828-46e2-92a8-d087ac774ba5"). InnerVolumeSpecName "kube-api-access-97hm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:54:05.886749 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.886711 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:05.934888 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.934849 2559 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3ecd7b12-c828-46e2-92a8-d087ac774ba5-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:54:05.934888 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:05.934882 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97hm9\" (UniqueName: \"kubernetes.io/projected/3ecd7b12-c828-46e2-92a8-d087ac774ba5-kube-api-access-97hm9\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:54:06.153761 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.153682 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-qx55p"] Sep 16 18:54:06.157026 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:54:06.157000 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1560d2_5746_4918_bcdb_8e95be0526ab.slice/crio-23a426e8cf85e227eb152c0ea59453811e7c43fe7374d8461242ff22f4c5840b WatchSource:0}: Error finding container 23a426e8cf85e227eb152c0ea59453811e7c43fe7374d8461242ff22f4c5840b: Status 404 returned error can't find the container with id 23a426e8cf85e227eb152c0ea59453811e7c43fe7374d8461242ff22f4c5840b Sep 16 18:54:06.608326 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.608300 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" event={"ID":"ee1560d2-5746-4918-bcdb-8e95be0526ab","Type":"ContainerStarted","Data":"23a426e8cf85e227eb152c0ea59453811e7c43fe7374d8461242ff22f4c5840b"} Sep 16 18:54:06.609475 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.609453 2559 generic.go:356] "Generic (PLEG): container finished" podID="3ecd7b12-c828-46e2-92a8-d087ac774ba5" containerID="cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a" exitCode=0 Sep 16 18:54:06.609578 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.609505 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" event={"ID":"3ecd7b12-c828-46e2-92a8-d087ac774ba5","Type":"ContainerDied","Data":"cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a"} Sep 16 18:54:06.609578 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.609518 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" Sep 16 18:54:06.609578 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.609539 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-cxgfb" event={"ID":"3ecd7b12-c828-46e2-92a8-d087ac774ba5","Type":"ContainerDied","Data":"5f8db5da66e942f2cae7a29117cb6902842da0bc35a6fd5a987731068195e844"} Sep 16 18:54:06.609578 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.609559 2559 scope.go:117] "RemoveContainer" containerID="cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a" Sep 16 18:54:06.617500 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.617477 2559 scope.go:117] "RemoveContainer" containerID="cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a" Sep 16 18:54:06.617822 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:54:06.617759 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a\": container with ID starting with cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a not found: ID does not exist" containerID="cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a" Sep 16 18:54:06.617822 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.617783 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a"} err="failed to get container status \"cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a\": rpc error: code = NotFound desc = could not find container \"cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a\": container with ID starting with cce43ee0b2653be2acb922d6b8e693ecb34079390c7063c589f63505571aa59a not found: ID does not exist" Sep 16 18:54:06.632082 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.632056 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-cxgfb"] Sep 16 18:54:06.635145 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.635124 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-75c4c6c467-cxgfb"] Sep 16 18:54:06.721918 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:06.721888 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ecd7b12-c828-46e2-92a8-d087ac774ba5" path="/var/lib/kubelet/pods/3ecd7b12-c828-46e2-92a8-d087ac774ba5/volumes" Sep 16 18:54:07.614304 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:07.614269 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" event={"ID":"ee1560d2-5746-4918-bcdb-8e95be0526ab","Type":"ContainerStarted","Data":"f43578e1dc094439e9cf1fd93a8e493a5643a461445bd9673b06acef6f192869"} Sep 16 18:54:07.614759 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:07.614345 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:07.637315 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:07.637270 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" podStartSLOduration=2.231156508 podStartE2EDuration="2.637256314s" totalImagesPullingTime="406.099806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:07.637139992 +0000 UTC m=+535.436261860" watchObservedRunningTime="2026-09-16 18:54:07.637256314 +0000 UTC m=+535.436378152" Sep 16 18:54:38.624115 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:38.624081 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-75c4c6c467-qx55p" Sep 16 18:54:39.999782 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:39.999749 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:40.044979 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.044937 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-n95hw"] Sep 16 18:54:40.045191 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.045179 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ecd7b12-c828-46e2-92a8-d087ac774ba5" containerName="manager" Sep 16 18:54:40.045232 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.045193 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ecd7b12-c828-46e2-92a8-d087ac774ba5" containerName="manager" Sep 16 18:54:40.045265 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.045239 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ecd7b12-c828-46e2-92a8-d087ac774ba5" containerName="manager" Sep 16 18:54:40.047788 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.047774 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:40.052144 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.052075 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 16 18:54:40.052144 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.052078 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4b6xj\"" Sep 16 18:54:40.059755 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.059730 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-n95hw"] Sep 16 18:54:40.093638 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.093613 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:40.174228 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.174193 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d-tls-certs\") pod \"model-serving-api-7b954ffc95-n95hw\" (UID: \"ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d\") " pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:40.174388 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.174254 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7nl5\" (UniqueName: \"kubernetes.io/projected/ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d-kube-api-access-p7nl5\") pod \"model-serving-api-7b954ffc95-n95hw\" (UID: \"ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d\") " pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:40.264402 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.264323 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:40.275323 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.275300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7nl5\" (UniqueName: \"kubernetes.io/projected/ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d-kube-api-access-p7nl5\") pod \"model-serving-api-7b954ffc95-n95hw\" (UID: \"ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d\") " pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:40.275425 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.275336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d-tls-certs\") pod \"model-serving-api-7b954ffc95-n95hw\" (UID: \"ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d\") " pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:40.277742 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.277719 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d-tls-certs\") pod \"model-serving-api-7b954ffc95-n95hw\" (UID: \"ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d\") " pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:40.284145 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.284122 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7nl5\" (UniqueName: \"kubernetes.io/projected/ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d-kube-api-access-p7nl5\") pod \"model-serving-api-7b954ffc95-n95hw\" (UID: \"ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d\") " pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:40.357749 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.357717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:40.363810 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.363773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:40.629513 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.629395 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-n95hw"] Sep 16 18:54:40.632114 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:54:40.632087 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6bff9d_15ba_4bd7_9e55_2f3917d77c8d.slice/crio-700542c49f44bb52df8e0fce07a902fb554e50b258982dd8ac7c02c7854ffd91 WatchSource:0}: Error finding container 700542c49f44bb52df8e0fce07a902fb554e50b258982dd8ac7c02c7854ffd91: Status 404 returned error can't find the container with id 700542c49f44bb52df8e0fce07a902fb554e50b258982dd8ac7c02c7854ffd91 Sep 16 18:54:40.727000 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:40.726955 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-n95hw" event={"ID":"ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d","Type":"ContainerStarted","Data":"700542c49f44bb52df8e0fce07a902fb554e50b258982dd8ac7c02c7854ffd91"} Sep 16 18:54:41.223035 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:41.223000 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:41.305899 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:41.305866 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:43.740221 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:43.740175 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-n95hw" event={"ID":"ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d","Type":"ContainerStarted","Data":"e5b9d9cac3f83ae130645a508993e42092850de3455fa5464051a23452382211"} Sep 16 18:54:43.740656 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:43.740330 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:54:43.763027 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:43.762959 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-n95hw" podStartSLOduration=1.538172114 podStartE2EDuration="3.76294386s" totalImagesPullingTime="2.224771746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:54:43.761938674 +0000 UTC m=+571.561060523" watchObservedRunningTime="2026-09-16 18:54:43.76294386 +0000 UTC m=+571.562065774" Sep 16 18:54:52.630024 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:52.629957 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:54:54.747634 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:54:54.747603 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-n95hw" Sep 16 18:55:12.614946 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:55:12.614915 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 18:55:12.615403 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:55:12.615197 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 18:56:15.536850 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:15.536807 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:15.600448 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:15.600419 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:19.698237 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:19.698196 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:27.476648 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.476614 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:27.616835 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.616803 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj"] Sep 16 18:56:27.619838 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.619823 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.623268 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.623245 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b26b5-kube-rbac-proxy-sar-config\"" Sep 16 18:56:27.623413 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.623399 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:56:27.624057 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.624036 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:56:27.624180 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.624077 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:56:27.624255 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.624150 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b26b5-predictor-serving-cert\"" Sep 16 18:56:27.624713 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.624696 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:56:27.624811 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.624715 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\"" Sep 16 18:56:27.637568 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.637549 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj"] Sep 16 18:56:27.700991 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.700950 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de0469a-5cbb-47d0-9830-5c940b4bf320-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.701212 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.701190 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclrb\" (UniqueName: \"kubernetes.io/projected/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-api-access-hclrb\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.701318 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.701227 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4de0469a-5cbb-47d0-9830-5c940b4bf320-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.701318 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.701277 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.701419 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.701403 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.702146 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.702124 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:27.802308 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.802233 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.802308 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.802278 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de0469a-5cbb-47d0-9830-5c940b4bf320-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.802308 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.802296 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclrb\" (UniqueName: \"kubernetes.io/projected/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-api-access-hclrb\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.802543 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.802314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4de0469a-5cbb-47d0-9830-5c940b4bf320-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.802543 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.802345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.802761 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.802738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4de0469a-5cbb-47d0-9830-5c940b4bf320-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.802995 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.802951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.802995 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.802990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.804814 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.804766 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:27.804910 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.804901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de0469a-5cbb-47d0-9830-5c940b4bf320-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.814607 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.814581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclrb\" (UniqueName: \"kubernetes.io/projected/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-api-access-hclrb\") pod \"isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.929443 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.929414 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:27.951087 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:27.951055 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:28.027877 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:28.027843 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:56:28.204109 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:28.204080 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj"] Sep 16 18:56:28.208116 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:56:28.208085 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de0469a_5cbb_47d0_9830_5c940b4bf320.slice/crio-01104eddde6dc7da7d956ce7ac16030cb8f91803bf06de6471c57604a9646ad7 WatchSource:0}: Error finding container 01104eddde6dc7da7d956ce7ac16030cb8f91803bf06de6471c57604a9646ad7: Status 404 returned error can't find the container with id 01104eddde6dc7da7d956ce7ac16030cb8f91803bf06de6471c57604a9646ad7 Sep 16 18:56:29.097755 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:29.097713 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerStarted","Data":"01104eddde6dc7da7d956ce7ac16030cb8f91803bf06de6471c57604a9646ad7"} Sep 16 18:56:32.111351 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:32.111315 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerStarted","Data":"452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2"} Sep 16 18:56:36.126305 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:36.126264 2559 generic.go:356] "Generic (PLEG): container finished" podID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerID="452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2" exitCode=0 Sep 16 18:56:36.126686 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:36.126337 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerDied","Data":"452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2"} Sep 16 18:56:49.183844 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:49.183809 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerStarted","Data":"f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b"} Sep 16 18:56:51.193361 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:51.193334 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerStarted","Data":"4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69"} Sep 16 18:56:54.208026 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:54.207993 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerStarted","Data":"41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb"} Sep 16 18:56:54.208430 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:54.208148 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:54.234270 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:54.234226 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podStartSLOduration=3.014849077 podStartE2EDuration="27.234211917s" totalImagesPullingTime="20.466317147s" totalInitContainerRuntime="3.753045693s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:27 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:54.232478125 +0000 UTC m=+702.031599985" watchObservedRunningTime="2026-09-16 18:56:54.234211917 +0000 UTC m=+702.033333757" Sep 16 18:56:55.211365 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:55.211333 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:55.211729 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:55.211392 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:55.212700 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:55.212665 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:56:55.213435 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:55.213408 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:56.218738 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:56.218695 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:56:56.219464 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:56.219424 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:56:56.223926 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:56.223904 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:56:57.220433 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:57.220394 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:56:57.220844 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:56:57.220652 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:02.389294 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:02.389236 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:07.221417 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:07.221314 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:57:07.221797 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:07.221631 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:17.220521 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:17.220476 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:57:17.221077 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:17.221038 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:21.474529 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:21.474494 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:22.560373 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:22.560338 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:57:27.221250 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:27.221209 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:57:27.221672 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:27.221642 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:37.221325 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:37.221274 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:57:37.221785 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:37.221667 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:47.221222 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:47.221165 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:57:47.221699 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:47.221675 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:57:57.220815 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:57.220768 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:57:57.221337 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:57:57.221315 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:58:07.221157 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:07.221126 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:58:07.221531 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:07.221216 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:58:17.649583 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:17.649490 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:58:22.662593 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.662557 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj"] Sep 16 18:58:22.663022 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.662833 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" containerID="cri-o://41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb" gracePeriod=30 Sep 16 18:58:22.663022 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.662866 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" containerID="cri-o://4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69" gracePeriod=30 Sep 16 18:58:22.663022 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.662833 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" containerID="cri-o://f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b" gracePeriod=30 Sep 16 18:58:22.689317 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.689285 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:58:22.813224 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.813184 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:58:22.959198 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.959158 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:58:22.990840 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.990811 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv"] Sep 16 18:58:22.992879 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.992863 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:22.995672 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.995649 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-f7202-predictor-serving-cert\"" Sep 16 18:58:22.999138 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:22.999109 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-f7202-kube-rbac-proxy-sar-config\"" Sep 16 18:58:23.010942 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.010919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv"] Sep 16 18:58:23.147288 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.147253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.147445 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.147294 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.147445 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.147396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwvj7\" (UniqueName: \"kubernetes.io/projected/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-api-access-pwvj7\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.147445 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.147438 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.147553 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.147470 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.230008 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.229905 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:58:23.247977 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.247942 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.248072 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.247991 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.248072 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.248032 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwvj7\" (UniqueName: \"kubernetes.io/projected/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-api-access-pwvj7\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.248072 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.248060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.248231 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.248078 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.248231 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:58:23.248184 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-serving-cert: secret "isvc-xgboost-graph-raw-f7202-predictor-serving-cert" not found Sep 16 18:58:23.248337 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:58:23.248257 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls podName:aff5faf1-bf8e-4f25-8ff8-c0c43df217d1 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:23.748237717 +0000 UTC m=+791.547359543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls") pod "isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" (UID: "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1") : secret "isvc-xgboost-graph-raw-f7202-predictor-serving-cert" not found Sep 16 18:58:23.248421 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.248401 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.248715 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.248695 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-cabundle-cert\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.248759 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.248743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.257486 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.257464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwvj7\" (UniqueName: \"kubernetes.io/projected/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-api-access-pwvj7\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.522091 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.522003 2559 generic.go:356] "Generic (PLEG): container finished" podID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerID="4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69" exitCode=2 Sep 16 18:58:23.522230 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.522088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerDied","Data":"4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69"} Sep 16 18:58:23.722907 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.722867 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:58:23.751981 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.751943 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.754265 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.754248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls\") pod \"isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:23.902437 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:23.902368 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:24.175939 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:24.175907 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv"] Sep 16 18:58:24.179781 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:58:24.179750 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff5faf1_bf8e_4f25_8ff8_c0c43df217d1.slice/crio-3cc13c01a6a944f3a363fd390b82a06b2fa6fa285c9ced8deccce6374b8f6390 WatchSource:0}: Error finding container 3cc13c01a6a944f3a363fd390b82a06b2fa6fa285c9ced8deccce6374b8f6390: Status 404 returned error can't find the container with id 3cc13c01a6a944f3a363fd390b82a06b2fa6fa285c9ced8deccce6374b8f6390 Sep 16 18:58:24.527942 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:24.527860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" event={"ID":"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1","Type":"ContainerStarted","Data":"a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f"} Sep 16 18:58:24.527942 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:24.527916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" event={"ID":"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1","Type":"ContainerStarted","Data":"3cc13c01a6a944f3a363fd390b82a06b2fa6fa285c9ced8deccce6374b8f6390"} Sep 16 18:58:26.217093 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:26.217048 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused" Sep 16 18:58:27.220343 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:27.220301 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:58:27.220748 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:27.220669 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:58:27.545630 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:27.545548 2559 generic.go:356] "Generic (PLEG): container finished" podID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerID="f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b" exitCode=0 Sep 16 18:58:27.545630 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:27.545603 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerDied","Data":"f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b"} Sep 16 18:58:28.550437 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:28.550358 2559 generic.go:356] "Generic (PLEG): container finished" podID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerID="a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f" exitCode=0 Sep 16 18:58:28.550771 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:28.550432 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" event={"ID":"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1","Type":"ContainerDied","Data":"a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f"} Sep 16 18:58:28.551798 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:28.551775 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:58:31.217450 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:31.217403 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused" Sep 16 18:58:36.218058 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:36.217927 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused" Sep 16 18:58:36.218471 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:36.218107 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:58:37.221213 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:37.221163 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:58:37.221661 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:37.221561 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:58:41.217281 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:41.217238 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused" Sep 16 18:58:46.217327 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:46.217277 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused" Sep 16 18:58:46.630437 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:46.630362 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" event={"ID":"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1","Type":"ContainerStarted","Data":"a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063"} Sep 16 18:58:46.630437 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:46.630422 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" event={"ID":"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1","Type":"ContainerStarted","Data":"1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74"} Sep 16 18:58:46.630712 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:46.630694 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:46.679933 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:46.679887 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podStartSLOduration=2.956310554 podStartE2EDuration="24.679872543s" totalImagesPullingTime="17.69035848s" totalInitContainerRuntime="4.033203509s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:46.679193153 +0000 UTC m=+814.478315024" watchObservedRunningTime="2026-09-16 18:58:46.679872543 +0000 UTC m=+814.478994382" Sep 16 18:58:47.220661 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:47.220617 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.18:8080: connect: connection refused" Sep 16 18:58:47.221121 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:47.220762 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:58:47.221121 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:47.220996 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 18:58:47.221214 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:47.221128 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:58:47.634582 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:47.634499 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:47.635881 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:47.635857 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:58:48.638145 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:48.638107 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:58:51.218104 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:51.218064 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.18:8643/healthz\": dial tcp 10.132.0.18:8643: connect: connection refused" Sep 16 18:58:52.800674 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.800652 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:58:52.877131 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.877100 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-cabundle-cert\") pod \"4de0469a-5cbb-47d0-9830-5c940b4bf320\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " Sep 16 18:58:52.877298 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.877137 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4de0469a-5cbb-47d0-9830-5c940b4bf320-kserve-provision-location\") pod \"4de0469a-5cbb-47d0-9830-5c940b4bf320\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " Sep 16 18:58:52.877298 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.877195 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de0469a-5cbb-47d0-9830-5c940b4bf320-proxy-tls\") pod \"4de0469a-5cbb-47d0-9830-5c940b4bf320\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " Sep 16 18:58:52.877298 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.877231 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hclrb\" (UniqueName: \"kubernetes.io/projected/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-api-access-hclrb\") pod \"4de0469a-5cbb-47d0-9830-5c940b4bf320\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " Sep 16 18:58:52.877298 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.877285 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-rbac-proxy-sar-config\") pod \"4de0469a-5cbb-47d0-9830-5c940b4bf320\" (UID: \"4de0469a-5cbb-47d0-9830-5c940b4bf320\") " Sep 16 18:58:52.877515 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.877485 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4de0469a-5cbb-47d0-9830-5c940b4bf320-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4de0469a-5cbb-47d0-9830-5c940b4bf320" (UID: "4de0469a-5cbb-47d0-9830-5c940b4bf320"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:58:52.877515 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.877507 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4de0469a-5cbb-47d0-9830-5c940b4bf320" (UID: "4de0469a-5cbb-47d0-9830-5c940b4bf320"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:58:52.877714 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.877692 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4de0469a-5cbb-47d0-9830-5c940b4bf320" (UID: "4de0469a-5cbb-47d0-9830-5c940b4bf320"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:58:52.879405 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.879386 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de0469a-5cbb-47d0-9830-5c940b4bf320-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4de0469a-5cbb-47d0-9830-5c940b4bf320" (UID: "4de0469a-5cbb-47d0-9830-5c940b4bf320"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:58:52.879496 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.879408 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-api-access-hclrb" (OuterVolumeSpecName: "kube-api-access-hclrb") pod "4de0469a-5cbb-47d0-9830-5c940b4bf320" (UID: "4de0469a-5cbb-47d0-9830-5c940b4bf320"). InnerVolumeSpecName "kube-api-access-hclrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:58:52.978258 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.978175 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4de0469a-5cbb-47d0-9830-5c940b4bf320-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:58:52.978258 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.978206 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hclrb\" (UniqueName: \"kubernetes.io/projected/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-api-access-hclrb\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:58:52.978258 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.978215 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:58:52.978258 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.978225 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4de0469a-5cbb-47d0-9830-5c940b4bf320-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:58:52.978258 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:52.978234 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4de0469a-5cbb-47d0-9830-5c940b4bf320-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 18:58:53.642340 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.642285 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:58:53.642882 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.642851 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:58:53.659267 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.659239 2559 generic.go:356] "Generic (PLEG): container finished" podID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerID="41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb" exitCode=0 Sep 16 18:58:53.659393 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.659295 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" Sep 16 18:58:53.659393 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.659324 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerDied","Data":"41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb"} Sep 16 18:58:53.659500 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.659398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj" event={"ID":"4de0469a-5cbb-47d0-9830-5c940b4bf320","Type":"ContainerDied","Data":"01104eddde6dc7da7d956ce7ac16030cb8f91803bf06de6471c57604a9646ad7"} Sep 16 18:58:53.659500 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.659447 2559 scope.go:117] "RemoveContainer" containerID="41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb" Sep 16 18:58:53.668220 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.668092 2559 scope.go:117] "RemoveContainer" containerID="4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69" Sep 16 18:58:53.675810 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.675790 2559 scope.go:117] "RemoveContainer" containerID="f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b" Sep 16 18:58:53.683460 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.683446 2559 scope.go:117] "RemoveContainer" containerID="452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2" Sep 16 18:58:53.687421 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.687399 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj"] Sep 16 18:58:53.691491 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.691470 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b26b5-predictor-844cf8fdf4-dhqlj"] Sep 16 18:58:53.691649 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.691630 2559 scope.go:117] "RemoveContainer" containerID="41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb" Sep 16 18:58:53.691941 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:58:53.691915 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb\": container with ID starting with 41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb not found: ID does not exist" containerID="41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb" Sep 16 18:58:53.692036 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.691949 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb"} err="failed to get container status \"41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb\": rpc error: code = NotFound desc = could not find container \"41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb\": container with ID starting with 41a041307a05a9b5cb128e1e7f812dbd86cf9ea47289953142e0311d18b9ccdb not found: ID does not exist" Sep 16 18:58:53.692036 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.691988 2559 scope.go:117] "RemoveContainer" containerID="4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69" Sep 16 18:58:53.692258 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:58:53.692241 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69\": container with ID starting with 4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69 not found: ID does not exist" containerID="4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69" Sep 16 18:58:53.692298 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.692266 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69"} err="failed to get container status \"4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69\": rpc error: code = NotFound desc = could not find container \"4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69\": container with ID starting with 4baaa00b42ac0737b386c3d1897f00a6f72087ce514492adeb9f35bb170e1b69 not found: ID does not exist" Sep 16 18:58:53.692298 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.692289 2559 scope.go:117] "RemoveContainer" containerID="f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b" Sep 16 18:58:53.692533 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:58:53.692520 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b\": container with ID starting with f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b not found: ID does not exist" containerID="f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b" Sep 16 18:58:53.692569 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.692538 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b"} err="failed to get container status \"f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b\": rpc error: code = NotFound desc = could not find container \"f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b\": container with ID starting with f670be8947cff2f21bb12b0fece27133010bec2ebac51c6a69ce5dbe526ed31b not found: ID does not exist" Sep 16 18:58:53.692569 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.692550 2559 scope.go:117] "RemoveContainer" containerID="452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2" Sep 16 18:58:53.692812 ip-10-0-140-230 kubenswrapper[2559]: E0916 18:58:53.692775 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2\": container with ID starting with 452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2 not found: ID does not exist" containerID="452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2" Sep 16 18:58:53.692866 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:53.692818 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2"} err="failed to get container status \"452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2\": rpc error: code = NotFound desc = could not find container \"452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2\": container with ID starting with 452287708cca9137bd01a042010da5ec289612049d247c0684207339dbcb47b2 not found: ID does not exist" Sep 16 18:58:54.721264 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:58:54.721233 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" path="/var/lib/kubelet/pods/4de0469a-5cbb-47d0-9830-5c940b4bf320/volumes" Sep 16 18:59:03.642855 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:03.642814 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:59:13.643178 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:13.643141 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:59:23.643196 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:23.643157 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:59:33.637219 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:33.637188 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:59:33.643215 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:33.643178 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:59:43.643764 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:43.643719 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 18:59:47.713354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:47.713319 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:59:52.843825 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.843785 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:59:52.914320 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914286 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8"] Sep 16 18:59:52.914567 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914555 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" Sep 16 18:59:52.914610 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914569 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" Sep 16 18:59:52.914610 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914579 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" Sep 16 18:59:52.914610 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914584 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" Sep 16 18:59:52.914610 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914596 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="storage-initializer" Sep 16 18:59:52.914610 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914601 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="storage-initializer" Sep 16 18:59:52.914610 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914611 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" Sep 16 18:59:52.914788 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914616 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" Sep 16 18:59:52.914788 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914658 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="agent" Sep 16 18:59:52.914788 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914669 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kserve-container" Sep 16 18:59:52.914788 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.914675 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4de0469a-5cbb-47d0-9830-5c940b4bf320" containerName="kube-rbac-proxy" Sep 16 18:59:52.916649 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.916607 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:52.919537 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.919515 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\"" Sep 16 18:59:52.919655 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.919558 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Sep 16 18:59:52.932542 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.932514 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8"] Sep 16 18:59:52.949902 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:52.949871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:59:53.007402 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.007370 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.007618 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.007411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.007618 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.007448 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.007618 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.007483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78l4\" (UniqueName: \"kubernetes.io/projected/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-api-access-d78l4\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.007618 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.007515 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.048657 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.048625 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:59:53.108762 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.108670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78l4\" (UniqueName: \"kubernetes.io/projected/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-api-access-d78l4\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.108762 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.108730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.109006 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.108778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.109006 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.108806 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.109006 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.108830 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.109234 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.109215 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.109623 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.109602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.109666 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.109602 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.111212 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.111194 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.117673 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.117646 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78l4\" (UniqueName: \"kubernetes.io/projected/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-api-access-d78l4\") pod \"isvc-canary-stop-predictor-57d5c586ff-vj8n8\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.162354 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.162317 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 18:59:53.226550 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.226504 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:53.498493 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.498460 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8"] Sep 16 18:59:53.501475 ip-10-0-140-230 kubenswrapper[2559]: W0916 18:59:53.501447 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47b81ac8_acf1_4aea_9b34_c42b9c3d41d6.slice/crio-fc2ef146ce2f9706522ad21f8558b394ac77116ef874a522d71ab3a1ea091072 WatchSource:0}: Error finding container fc2ef146ce2f9706522ad21f8558b394ac77116ef874a522d71ab3a1ea091072: Status 404 returned error can't find the container with id fc2ef146ce2f9706522ad21f8558b394ac77116ef874a522d71ab3a1ea091072 Sep 16 18:59:53.643593 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.643563 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 18:59:53.883760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.883673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" event={"ID":"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6","Type":"ContainerStarted","Data":"f380e6e06a7b26939ffc4f9f109f63a3638e977b4b0598856e43185ca760d301"} Sep 16 18:59:53.883760 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:53.883736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" event={"ID":"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6","Type":"ContainerStarted","Data":"fc2ef146ce2f9706522ad21f8558b394ac77116ef874a522d71ab3a1ea091072"} Sep 16 18:59:57.899551 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:57.899517 2559 generic.go:356] "Generic (PLEG): container finished" podID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerID="f380e6e06a7b26939ffc4f9f109f63a3638e977b4b0598856e43185ca760d301" exitCode=0 Sep 16 18:59:57.899936 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:57.899570 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" event={"ID":"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6","Type":"ContainerDied","Data":"f380e6e06a7b26939ffc4f9f109f63a3638e977b4b0598856e43185ca760d301"} Sep 16 18:59:58.905195 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:58.905165 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" event={"ID":"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6","Type":"ContainerStarted","Data":"31b86ab6d7d7a6bb3af65ae090252b260e5fee1503927b4d2bd48145d09f1f12"} Sep 16 18:59:58.905609 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:58.905233 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" event={"ID":"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6","Type":"ContainerStarted","Data":"f820b31ebc187ab95127a81bb27cfbfa1c412fe8455b70e54e31fb115f582191"} Sep 16 18:59:58.905609 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:58.905418 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:58.934352 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:58.934302 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podStartSLOduration=3.231341415 podStartE2EDuration="6.934284163s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.702942748s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:59:58.932119988 +0000 UTC m=+886.731241842" watchObservedRunningTime="2026-09-16 18:59:58.934284163 +0000 UTC m=+886.733406001" Sep 16 18:59:59.908674 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:59.908644 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 18:59:59.909903 ip-10-0-140-230 kubenswrapper[2559]: I0916 18:59:59.909878 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:00:00.912382 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:00.912340 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:00:02.975258 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:02.975215 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:00:03.438852 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:03.438816 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:00:05.917297 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:05.917224 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 19:00:05.917751 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:05.917719 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:00:12.638555 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:12.638525 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 19:00:12.638991 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:12.638642 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 19:00:12.915752 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:12.915723 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:00:12.954300 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:12.954270 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv"] Sep 16 19:00:12.954575 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:12.954548 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" containerID="cri-o://1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74" gracePeriod=30 Sep 16 19:00:12.959916 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:12.959880 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kube-rbac-proxy" containerID="cri-o://a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063" gracePeriod=30 Sep 16 19:00:13.027318 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.027281 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:00:13.149245 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.149215 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:00:13.308617 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.308538 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:00:13.595948 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.595863 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:00:13.638520 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.638472 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.19:8643/healthz\": dial tcp 10.132.0.19:8643: connect: connection refused" Sep 16 19:00:13.643800 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.643778 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.19:8080: connect: connection refused" Sep 16 19:00:13.696287 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.696259 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:00:13.961501 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.961470 2559 generic.go:356] "Generic (PLEG): container finished" podID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerID="a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063" exitCode=2 Sep 16 19:00:13.961664 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:13.961521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" event={"ID":"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1","Type":"ContainerDied","Data":"a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063"} Sep 16 19:00:15.917780 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:15.917744 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:00:16.801355 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.801330 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 19:00:16.886204 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.886122 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls\") pod \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " Sep 16 19:00:16.886204 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.886159 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-cabundle-cert\") pod \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " Sep 16 19:00:16.886204 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.886192 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-rbac-proxy-sar-config\") pod \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " Sep 16 19:00:16.886506 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.886209 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwvj7\" (UniqueName: \"kubernetes.io/projected/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-api-access-pwvj7\") pod \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " Sep 16 19:00:16.886506 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.886353 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kserve-provision-location\") pod \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\" (UID: \"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1\") " Sep 16 19:00:16.886614 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.886514 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" (UID: "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:00:16.886708 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.886682 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" (UID: "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:00:16.886708 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.886695 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" (UID: "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:00:16.888275 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.888252 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" (UID: "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:00:16.888362 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.888335 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-api-access-pwvj7" (OuterVolumeSpecName: "kube-api-access-pwvj7") pod "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" (UID: "aff5faf1-bf8e-4f25-8ff8-c0c43df217d1"). InnerVolumeSpecName "kube-api-access-pwvj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:00:16.973423 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.973381 2559 generic.go:356] "Generic (PLEG): container finished" podID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerID="1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74" exitCode=0 Sep 16 19:00:16.973801 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.973442 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" Sep 16 19:00:16.973801 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.973469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" event={"ID":"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1","Type":"ContainerDied","Data":"1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74"} Sep 16 19:00:16.973801 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.973531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv" event={"ID":"aff5faf1-bf8e-4f25-8ff8-c0c43df217d1","Type":"ContainerDied","Data":"3cc13c01a6a944f3a363fd390b82a06b2fa6fa285c9ced8deccce6374b8f6390"} Sep 16 19:00:16.973801 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.973557 2559 scope.go:117] "RemoveContainer" containerID="a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063" Sep 16 19:00:16.982070 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.982041 2559 scope.go:117] "RemoveContainer" containerID="1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74" Sep 16 19:00:16.986827 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.986808 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:00:16.986901 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.986829 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:00:16.986901 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.986838 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:00:16.986901 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.986847 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwvj7\" (UniqueName: \"kubernetes.io/projected/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kube-api-access-pwvj7\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:00:16.986901 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.986856 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:00:16.988833 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.988816 2559 scope.go:117] "RemoveContainer" containerID="a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f" Sep 16 19:00:16.995473 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.995456 2559 scope.go:117] "RemoveContainer" containerID="a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063" Sep 16 19:00:16.995752 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:00:16.995732 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063\": container with ID starting with a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063 not found: ID does not exist" containerID="a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063" Sep 16 19:00:16.995817 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.995764 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063"} err="failed to get container status \"a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063\": rpc error: code = NotFound desc = could not find container \"a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063\": container with ID starting with a8c3413edb776b95bc78939517393737ed8032fa51945f7292132fbf2b623063 not found: ID does not exist" Sep 16 19:00:16.995817 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.995788 2559 scope.go:117] "RemoveContainer" containerID="1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74" Sep 16 19:00:16.996166 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:00:16.996103 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74\": container with ID starting with 1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74 not found: ID does not exist" containerID="1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74" Sep 16 19:00:16.996166 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.996153 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74"} err="failed to get container status \"1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74\": rpc error: code = NotFound desc = could not find container \"1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74\": container with ID starting with 1189ca499634100eab19942b93659b48b906b5aa6bc8656b472ceb11d3123e74 not found: ID does not exist" Sep 16 19:00:16.996343 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:16.996175 2559 scope.go:117] "RemoveContainer" containerID="a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f" Sep 16 19:00:17.000621 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:00:17.000594 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f\": container with ID starting with a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f not found: ID does not exist" containerID="a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f" Sep 16 19:00:17.000723 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:17.000626 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f"} err="failed to get container status \"a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f\": rpc error: code = NotFound desc = could not find container \"a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f\": container with ID starting with a18f0388b49e7523982c938512eb1c1e17675a3cef81b4dbea7123bce1f07c3f not found: ID does not exist" Sep 16 19:00:17.002268 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:17.002248 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv"] Sep 16 19:00:17.003229 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:17.003211 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-f7202-predictor-855ccc46fc-qkzbv"] Sep 16 19:00:18.720956 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:18.720926 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" path="/var/lib/kubelet/pods/aff5faf1-bf8e-4f25-8ff8-c0c43df217d1/volumes" Sep 16 19:00:25.918569 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:25.918526 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:00:35.918198 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:35.918151 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:00:45.918644 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:45.918605 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:00:55.918047 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:00:55.918008 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:01:05.918363 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:05.918317 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:01:15.918776 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:15.918745 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 19:01:32.920650 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:32.920620 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8"] Sep 16 19:01:32.921158 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:32.920844 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" containerID="cri-o://f820b31ebc187ab95127a81bb27cfbfa1c412fe8455b70e54e31fb115f582191" gracePeriod=30 Sep 16 19:01:32.921158 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:32.920882 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kube-rbac-proxy" containerID="cri-o://31b86ab6d7d7a6bb3af65ae090252b260e5fee1503927b4d2bd48145d09f1f12" gracePeriod=30 Sep 16 19:01:32.928895 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:32.928869 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:01:33.068991 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:33.068951 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:01:33.244709 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:33.244632 2559 generic.go:356] "Generic (PLEG): container finished" podID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerID="31b86ab6d7d7a6bb3af65ae090252b260e5fee1503927b4d2bd48145d09f1f12" exitCode=2 Sep 16 19:01:33.244709 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:33.244705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" event={"ID":"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6","Type":"ContainerDied","Data":"31b86ab6d7d7a6bb3af65ae090252b260e5fee1503927b4d2bd48145d09f1f12"} Sep 16 19:01:35.913132 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:35.913042 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.20:8643/healthz\": dial tcp 10.132.0.20:8643: connect: connection refused" Sep 16 19:01:35.918400 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:35.918368 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.20:8080: connect: connection refused" Sep 16 19:01:37.261807 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.261763 2559 generic.go:356] "Generic (PLEG): container finished" podID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerID="f820b31ebc187ab95127a81bb27cfbfa1c412fe8455b70e54e31fb115f582191" exitCode=0 Sep 16 19:01:37.262219 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.261838 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" event={"ID":"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6","Type":"ContainerDied","Data":"f820b31ebc187ab95127a81bb27cfbfa1c412fe8455b70e54e31fb115f582191"} Sep 16 19:01:37.361435 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.361410 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 19:01:37.481385 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.481306 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d78l4\" (UniqueName: \"kubernetes.io/projected/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-api-access-d78l4\") pod \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " Sep 16 19:01:37.481385 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.481356 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-proxy-tls\") pod \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " Sep 16 19:01:37.481600 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.481386 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kserve-provision-location\") pod \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " Sep 16 19:01:37.481600 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.481414 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-cabundle-cert\") pod \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " Sep 16 19:01:37.481600 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.481442 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-rbac-proxy-sar-config\") pod \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\" (UID: \"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6\") " Sep 16 19:01:37.481801 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.481771 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" (UID: "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:01:37.481905 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.481718 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" (UID: "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:01:37.481905 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.481877 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" (UID: "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:01:37.483460 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.483435 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-api-access-d78l4" (OuterVolumeSpecName: "kube-api-access-d78l4") pod "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" (UID: "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6"). InnerVolumeSpecName "kube-api-access-d78l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:01:37.483538 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.483448 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" (UID: "47b81ac8-acf1-4aea-9b34-c42b9c3d41d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:01:37.582370 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.582314 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78l4\" (UniqueName: \"kubernetes.io/projected/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-api-access-d78l4\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:01:37.582370 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.582362 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:01:37.582370 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.582375 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:01:37.582370 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.582385 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:01:37.582630 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:37.582394 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:01:38.267537 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:38.267500 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" event={"ID":"47b81ac8-acf1-4aea-9b34-c42b9c3d41d6","Type":"ContainerDied","Data":"fc2ef146ce2f9706522ad21f8558b394ac77116ef874a522d71ab3a1ea091072"} Sep 16 19:01:38.267537 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:38.267518 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8" Sep 16 19:01:38.268085 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:38.267566 2559 scope.go:117] "RemoveContainer" containerID="31b86ab6d7d7a6bb3af65ae090252b260e5fee1503927b4d2bd48145d09f1f12" Sep 16 19:01:38.275600 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:38.275566 2559 scope.go:117] "RemoveContainer" containerID="f820b31ebc187ab95127a81bb27cfbfa1c412fe8455b70e54e31fb115f582191" Sep 16 19:01:38.282407 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:38.282386 2559 scope.go:117] "RemoveContainer" containerID="f380e6e06a7b26939ffc4f9f109f63a3638e977b4b0598856e43185ca760d301" Sep 16 19:01:38.295870 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:38.295848 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8"] Sep 16 19:01:38.300455 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:38.300431 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-vj8n8"] Sep 16 19:01:38.721448 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:38.721417 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" path="/var/lib/kubelet/pods/47b81ac8-acf1-4aea-9b34-c42b9c3d41d6/volumes" Sep 16 19:01:43.095634 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.095593 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:01:43.119033 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119003 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd"] Sep 16 19:01:43.119269 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119254 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kube-rbac-proxy" Sep 16 19:01:43.119310 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119272 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kube-rbac-proxy" Sep 16 19:01:43.119310 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119281 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" Sep 16 19:01:43.119310 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119288 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" Sep 16 19:01:43.119310 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119295 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kube-rbac-proxy" Sep 16 19:01:43.119310 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119300 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kube-rbac-proxy" Sep 16 19:01:43.119310 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119307 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119313 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119322 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="storage-initializer" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119328 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="storage-initializer" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119339 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="storage-initializer" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119347 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="storage-initializer" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119401 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kserve-container" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119407 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kserve-container" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119418 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="47b81ac8-acf1-4aea-9b34-c42b9c3d41d6" containerName="kube-rbac-proxy" Sep 16 19:01:43.119486 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.119426 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff5faf1-bf8e-4f25-8ff8-c0c43df217d1" containerName="kube-rbac-proxy" Sep 16 19:01:43.122245 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.122227 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.125223 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.125202 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9d9a89-predictor-serving-cert\"" Sep 16 19:01:43.125324 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.125250 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 19:01:43.125384 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.125338 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 19:01:43.125423 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.125396 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-9d9a89-kube-rbac-proxy-sar-config\"" Sep 16 19:01:43.125472 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.125457 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 19:01:43.125522 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.125509 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\"" Sep 16 19:01:43.126728 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.126710 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:01:43.134860 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.134837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd"] Sep 16 19:01:43.225566 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.225535 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.225710 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.225573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f54ln\" (UniqueName: \"kubernetes.io/projected/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-api-access-f54ln\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.225710 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.225601 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kserve-provision-location\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.225710 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.225631 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-cabundle-cert\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.225710 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.225658 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.326241 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.326213 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.326383 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.326247 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f54ln\" (UniqueName: \"kubernetes.io/projected/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-api-access-f54ln\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.326383 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.326265 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kserve-provision-location\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.326383 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.326295 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-cabundle-cert\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.326383 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.326333 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.326555 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:01:43.326374 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-serving-cert: secret "isvc-primary-9d9a89-predictor-serving-cert" not found Sep 16 19:01:43.326555 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:01:43.326445 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls podName:0cbfecba-d5da-4c61-aa73-d780c62d87d1 nodeName:}" failed. No retries permitted until 2026-09-16 19:01:43.826424143 +0000 UTC m=+991.625545974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls") pod "isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" (UID: "0cbfecba-d5da-4c61-aa73-d780c62d87d1") : secret "isvc-primary-9d9a89-predictor-serving-cert" not found Sep 16 19:01:43.326693 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.326672 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kserve-provision-location\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.326945 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.326930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-rbac-proxy-sar-config\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.327012 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.326941 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-cabundle-cert\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.336091 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.336072 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f54ln\" (UniqueName: \"kubernetes.io/projected/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-api-access-f54ln\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.483314 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.483281 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:01:43.828647 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.828570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:43.831066 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:43.831039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls\") pod \"isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:44.032079 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:44.032041 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:44.308423 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:44.308396 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd"] Sep 16 19:01:44.310880 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:01:44.310855 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cbfecba_d5da_4c61_aa73_d780c62d87d1.slice/crio-2aca6909d161d115d2fd3840d249a9a22ff145d7e1ca593e9bf3ab007de4ad63 WatchSource:0}: Error finding container 2aca6909d161d115d2fd3840d249a9a22ff145d7e1ca593e9bf3ab007de4ad63: Status 404 returned error can't find the container with id 2aca6909d161d115d2fd3840d249a9a22ff145d7e1ca593e9bf3ab007de4ad63 Sep 16 19:01:45.297689 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:45.297652 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" event={"ID":"0cbfecba-d5da-4c61-aa73-d780c62d87d1","Type":"ContainerStarted","Data":"4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67"} Sep 16 19:01:45.297845 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:45.297708 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" event={"ID":"0cbfecba-d5da-4c61-aa73-d780c62d87d1","Type":"ContainerStarted","Data":"2aca6909d161d115d2fd3840d249a9a22ff145d7e1ca593e9bf3ab007de4ad63"} Sep 16 19:01:48.309521 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:48.309487 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerID="4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67" exitCode=0 Sep 16 19:01:48.309906 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:48.309536 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" event={"ID":"0cbfecba-d5da-4c61-aa73-d780c62d87d1","Type":"ContainerDied","Data":"4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67"} Sep 16 19:01:49.314912 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:49.314880 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" event={"ID":"0cbfecba-d5da-4c61-aa73-d780c62d87d1","Type":"ContainerStarted","Data":"e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39"} Sep 16 19:01:49.315308 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:49.314944 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" event={"ID":"0cbfecba-d5da-4c61-aa73-d780c62d87d1","Type":"ContainerStarted","Data":"896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d"} Sep 16 19:01:49.315308 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:49.315156 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:49.341215 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:49.341169 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podStartSLOduration=2.515152466 podStartE2EDuration="6.34115427s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.826001804s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:49.339816724 +0000 UTC m=+997.138938581" watchObservedRunningTime="2026-09-16 19:01:49.34115427 +0000 UTC m=+997.140276108" Sep 16 19:01:50.318446 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:50.318400 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:50.319973 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:50.319939 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:01:51.321628 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:51.321591 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:01:56.326616 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:56.326587 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:01:56.327252 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:01:56.327220 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:02:03.212825 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:03.212787 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:03.324976 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:03.324942 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:06.327397 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:06.327357 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:02:13.326367 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.326333 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:13.398786 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.398756 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5"] Sep 16 19:02:13.403087 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.403065 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:13.406278 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.406256 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-2d3bd-predictor-serving-cert\"" Sep 16 19:02:13.406655 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.406635 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-2d3bd-kube-rbac-proxy-sar-config\"" Sep 16 19:02:13.418346 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.418323 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5"] Sep 16 19:02:13.437464 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.437436 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxllz\" (UniqueName: \"kubernetes.io/projected/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-api-access-lxllz\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:13.437464 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.437472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:13.437678 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.437558 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:13.537937 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.537906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:13.538110 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.537951 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxllz\" (UniqueName: \"kubernetes.io/projected/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-api-access-lxllz\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:13.538110 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.538075 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:13.538240 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:02:13.538223 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-serving-cert: secret "message-dumper-raw-2d3bd-predictor-serving-cert" not found Sep 16 19:02:13.538309 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:02:13.538300 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls podName:b2e9d1eb-3932-44a0-9bbb-f023649c868d nodeName:}" failed. No retries permitted until 2026-09-16 19:02:14.038281568 +0000 UTC m=+1021.837403403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls") pod "message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" (UID: "b2e9d1eb-3932-44a0-9bbb-f023649c868d") : secret "message-dumper-raw-2d3bd-predictor-serving-cert" not found Sep 16 19:02:13.538584 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.538566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:13.550414 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:13.550393 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxllz\" (UniqueName: \"kubernetes.io/projected/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-api-access-lxllz\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:14.042827 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:14.042792 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:14.045111 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:14.045089 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls\") pod \"message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:14.313158 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:14.313071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:14.587354 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:14.587276 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5"] Sep 16 19:02:14.589293 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:02:14.589266 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e9d1eb_3932_44a0_9bbb_f023649c868d.slice/crio-175cbf604f726e9aeb70fb2dfc9d60ee06649ad859651c3eb14bb7e1dab72df4 WatchSource:0}: Error finding container 175cbf604f726e9aeb70fb2dfc9d60ee06649ad859651c3eb14bb7e1dab72df4: Status 404 returned error can't find the container with id 175cbf604f726e9aeb70fb2dfc9d60ee06649ad859651c3eb14bb7e1dab72df4 Sep 16 19:02:14.951555 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:14.951513 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:15.408116 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:15.408042 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" event={"ID":"b2e9d1eb-3932-44a0-9bbb-f023649c868d","Type":"ContainerStarted","Data":"175cbf604f726e9aeb70fb2dfc9d60ee06649ad859651c3eb14bb7e1dab72df4"} Sep 16 19:02:16.327449 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:16.327409 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:02:16.413525 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:16.413486 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" event={"ID":"b2e9d1eb-3932-44a0-9bbb-f023649c868d","Type":"ContainerStarted","Data":"3796abaee6215aa6a1623a86ff921bf835081c0716345123b885d4a4af566a31"} Sep 16 19:02:16.413713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:16.413574 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" event={"ID":"b2e9d1eb-3932-44a0-9bbb-f023649c868d","Type":"ContainerStarted","Data":"2ed410c8eed207039b46884fbafb0bf643a2eb46db6ff39456a9c2c4be97f5a0"} Sep 16 19:02:16.413713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:16.413610 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:16.413713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:16.413638 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:16.415431 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:16.415410 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:16.433548 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:16.433501 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" podStartSLOduration=2.261391656 podStartE2EDuration="3.433486371s" totalImagesPullingTime="1.172094715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:16.432163215 +0000 UTC m=+1024.231285100" watchObservedRunningTime="2026-09-16 19:02:16.433486371 +0000 UTC m=+1024.232608210" Sep 16 19:02:23.426571 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:23.426542 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:02:26.327719 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:26.327681 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:02:33.432255 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.432222 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:33.467825 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.467796 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh"] Sep 16 19:02:33.471535 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.471515 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.474621 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.474602 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-2d3bd-predictor-serving-cert\"" Sep 16 19:02:33.474719 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.474641 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-2d3bd-kube-rbac-proxy-sar-config\"" Sep 16 19:02:33.489602 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.489579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh"] Sep 16 19:02:33.525789 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.525762 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:33.584564 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.584532 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10933a9d-e7fd-4b1a-9eda-a21f48f74639-proxy-tls\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.584721 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.584585 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.584721 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.584605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lts59\" (UniqueName: \"kubernetes.io/projected/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-api-access-lts59\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.584721 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.584690 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-cabundle-cert\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.584860 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.584723 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kserve-provision-location\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.685846 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.685770 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.685846 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.685808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lts59\" (UniqueName: \"kubernetes.io/projected/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-api-access-lts59\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.685846 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.685834 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-cabundle-cert\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.686175 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.685945 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kserve-provision-location\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.686175 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.686031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10933a9d-e7fd-4b1a-9eda-a21f48f74639-proxy-tls\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.686353 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.686331 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kserve-provision-location\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.686475 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.686458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.686571 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.686552 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-cabundle-cert\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.688430 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.688410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10933a9d-e7fd-4b1a-9eda-a21f48f74639-proxy-tls\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.695542 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.695521 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lts59\" (UniqueName: \"kubernetes.io/projected/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-api-access-lts59\") pod \"isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:33.781803 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:33.781769 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:34.057346 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:34.057284 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh"] Sep 16 19:02:34.059412 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:02:34.059383 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10933a9d_e7fd_4b1a_9eda_a21f48f74639.slice/crio-81c21f0ce82786fb778e4e508267f9f7bbb31a5c9da2ec69589524b02fff62a2 WatchSource:0}: Error finding container 81c21f0ce82786fb778e4e508267f9f7bbb31a5c9da2ec69589524b02fff62a2: Status 404 returned error can't find the container with id 81c21f0ce82786fb778e4e508267f9f7bbb31a5c9da2ec69589524b02fff62a2 Sep 16 19:02:34.478889 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:34.478854 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerStarted","Data":"b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322"} Sep 16 19:02:34.479279 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:34.478915 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerStarted","Data":"81c21f0ce82786fb778e4e508267f9f7bbb31a5c9da2ec69589524b02fff62a2"} Sep 16 19:02:36.327558 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:36.327520 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:02:38.495893 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:38.495856 2559 generic.go:356] "Generic (PLEG): container finished" podID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerID="b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322" exitCode=0 Sep 16 19:02:38.496315 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:38.495899 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerDied","Data":"b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322"} Sep 16 19:02:39.502211 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:39.502178 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerStarted","Data":"8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb"} Sep 16 19:02:39.502568 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:39.502243 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerStarted","Data":"03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078"} Sep 16 19:02:39.502568 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:39.502266 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerStarted","Data":"257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71"} Sep 16 19:02:39.502639 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:39.502591 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:39.502639 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:39.502635 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:39.503874 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:39.503850 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:02:39.532888 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:39.532840 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podStartSLOduration=2.571076701 podStartE2EDuration="6.532826006s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.961749305s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:02:39.530785521 +0000 UTC m=+1047.329907394" watchObservedRunningTime="2026-09-16 19:02:39.532826006 +0000 UTC m=+1047.331947845" Sep 16 19:02:40.506203 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:40.506168 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:40.506585 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:40.506276 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:02:40.507369 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:40.507342 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:02:41.510498 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:41.510453 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:02:41.510956 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:41.510933 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:02:43.941553 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:43.941518 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:02:46.327413 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:46.327374 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:02:46.515367 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:46.515333 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:02:46.516051 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:46.516009 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:02:46.516402 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:46.516370 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:02:56.327538 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:56.327495 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.21:8080: connect: connection refused" Sep 16 19:02:56.516753 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:56.516700 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:02:56.517244 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:02:56.517214 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:03:04.720559 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:04.720531 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:03:06.516218 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:06.516167 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:03:06.516637 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:06.516615 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:03:16.516135 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:16.516089 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:03:16.516668 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:16.516646 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:03:26.516268 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:26.516226 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:03:26.516737 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:26.516716 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:03:28.279232 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:28.279201 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:28.437342 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:28.437313 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:28.848592 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:28.848557 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:28.961805 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:28.961770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:36.516264 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:36.516222 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:03:36.516783 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:36.516724 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:03:44.617078 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:44.617021 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:46.516422 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.516382 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:03:46.516909 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.516875 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:03:46.560567 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.560532 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:46.561273 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.561253 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd"] Sep 16 19:03:46.561495 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.561458 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" containerID="cri-o://896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d" gracePeriod=30 Sep 16 19:03:46.561564 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.561494 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kube-rbac-proxy" containerID="cri-o://e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39" gracePeriod=30 Sep 16 19:03:46.633130 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.633091 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:46.751925 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.751896 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerID="e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39" exitCode=2 Sep 16 19:03:46.752088 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:46.752000 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" event={"ID":"0cbfecba-d5da-4c61-aa73-d780c62d87d1","Type":"ContainerDied","Data":"e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39"} Sep 16 19:03:48.717054 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:48.717012 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:03:48.717465 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:48.717394 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:03:51.199009 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.198986 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:03:51.250643 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.250613 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-cabundle-cert\") pod \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " Sep 16 19:03:51.250800 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.250675 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f54ln\" (UniqueName: \"kubernetes.io/projected/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-api-access-f54ln\") pod \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " Sep 16 19:03:51.250800 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.250697 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls\") pod \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " Sep 16 19:03:51.250800 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.250715 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kserve-provision-location\") pod \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " Sep 16 19:03:51.250800 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.250751 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-rbac-proxy-sar-config\") pod \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\" (UID: \"0cbfecba-d5da-4c61-aa73-d780c62d87d1\") " Sep 16 19:03:51.251045 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.251001 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0cbfecba-d5da-4c61-aa73-d780c62d87d1" (UID: "0cbfecba-d5da-4c61-aa73-d780c62d87d1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:03:51.251122 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.251098 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0cbfecba-d5da-4c61-aa73-d780c62d87d1" (UID: "0cbfecba-d5da-4c61-aa73-d780c62d87d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:03:51.251209 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.251190 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0cbfecba-d5da-4c61-aa73-d780c62d87d1" (UID: "0cbfecba-d5da-4c61-aa73-d780c62d87d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:03:51.252871 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.252842 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0cbfecba-d5da-4c61-aa73-d780c62d87d1" (UID: "0cbfecba-d5da-4c61-aa73-d780c62d87d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:03:51.252995 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.252847 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-api-access-f54ln" (OuterVolumeSpecName: "kube-api-access-f54ln") pod "0cbfecba-d5da-4c61-aa73-d780c62d87d1" (UID: "0cbfecba-d5da-4c61-aa73-d780c62d87d1"). InnerVolumeSpecName "kube-api-access-f54ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:03:51.351424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.351343 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f54ln\" (UniqueName: \"kubernetes.io/projected/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-api-access-f54ln\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:03:51.351424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.351372 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cbfecba-d5da-4c61-aa73-d780c62d87d1-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:03:51.351424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.351382 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:03:51.351424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.351390 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:03:51.351424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.351401 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cbfecba-d5da-4c61-aa73-d780c62d87d1-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:03:51.696761 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.696730 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:51.777244 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.777202 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerID="896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d" exitCode=0 Sep 16 19:03:51.777424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.777247 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" event={"ID":"0cbfecba-d5da-4c61-aa73-d780c62d87d1","Type":"ContainerDied","Data":"896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d"} Sep 16 19:03:51.777424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.777318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" event={"ID":"0cbfecba-d5da-4c61-aa73-d780c62d87d1","Type":"ContainerDied","Data":"2aca6909d161d115d2fd3840d249a9a22ff145d7e1ca593e9bf3ab007de4ad63"} Sep 16 19:03:51.777424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.777259 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd" Sep 16 19:03:51.777424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.777366 2559 scope.go:117] "RemoveContainer" containerID="e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39" Sep 16 19:03:51.786473 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.786455 2559 scope.go:117] "RemoveContainer" containerID="896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d" Sep 16 19:03:51.793696 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.793678 2559 scope.go:117] "RemoveContainer" containerID="4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67" Sep 16 19:03:51.796638 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.796595 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:51.800834 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.800808 2559 scope.go:117] "RemoveContainer" containerID="e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39" Sep 16 19:03:51.801137 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:03:51.801109 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39\": container with ID starting with e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39 not found: ID does not exist" containerID="e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39" Sep 16 19:03:51.801213 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.801145 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39"} err="failed to get container status \"e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39\": rpc error: code = NotFound desc = could not find container \"e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39\": container with ID starting with e6d082763226713d6739d861797407cfc1b9ef6896c5e28f314295cf1015bd39 not found: ID does not exist" Sep 16 19:03:51.801213 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.801167 2559 scope.go:117] "RemoveContainer" containerID="896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d" Sep 16 19:03:51.801437 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:03:51.801420 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d\": container with ID starting with 896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d not found: ID does not exist" containerID="896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d" Sep 16 19:03:51.801499 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.801446 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d"} err="failed to get container status \"896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d\": rpc error: code = NotFound desc = could not find container \"896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d\": container with ID starting with 896bb896c7039ad7728bb2cc6d4f4e16c5ddac0b2b7320a74032b67c12ed415d not found: ID does not exist" Sep 16 19:03:51.801499 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.801469 2559 scope.go:117] "RemoveContainer" containerID="4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67" Sep 16 19:03:51.801723 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:03:51.801704 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67\": container with ID starting with 4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67 not found: ID does not exist" containerID="4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67" Sep 16 19:03:51.801763 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.801729 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67"} err="failed to get container status \"4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67\": rpc error: code = NotFound desc = could not find container \"4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67\": container with ID starting with 4f60807eff01be79e289b490f67448bf763fd5b3e91df98908a6353003dedc67 not found: ID does not exist" Sep 16 19:03:51.802253 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.802204 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd"] Sep 16 19:03:51.803909 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:51.803892 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-9d9a89-predictor-6bdfff7bbc-z6vhd"] Sep 16 19:03:52.722113 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:52.722083 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" path="/var/lib/kubelet/pods/0cbfecba-d5da-4c61-aa73-d780c62d87d1/volumes" Sep 16 19:03:56.707577 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:56.707546 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:03:58.721129 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:58.721102 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:03:58.721463 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:03:58.721181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:04:01.909685 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:01.909650 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:04:02.024738 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:02.024706 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:04:08.439738 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.439704 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5_b2e9d1eb-3932-44a0-9bbb-f023649c868d/kserve-container/0.log" Sep 16 19:04:08.609118 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.609087 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh"] Sep 16 19:04:08.609372 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.609344 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" containerID="cri-o://257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71" gracePeriod=30 Sep 16 19:04:08.609501 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.609354 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" containerID="cri-o://8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb" gracePeriod=30 Sep 16 19:04:08.609574 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.609360 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" containerID="cri-o://03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078" gracePeriod=30 Sep 16 19:04:08.651462 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.651429 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5"] Sep 16 19:04:08.651748 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.651699 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerName="kserve-container" containerID="cri-o://2ed410c8eed207039b46884fbafb0bf643a2eb46db6ff39456a9c2c4be97f5a0" gracePeriod=30 Sep 16 19:04:08.651748 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.651728 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerName="kube-rbac-proxy" containerID="cri-o://3796abaee6215aa6a1623a86ff921bf835081c0716345123b885d4a4af566a31" gracePeriod=30 Sep 16 19:04:08.655318 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.655291 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:04:08.717098 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.716953 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:04:08.717373 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.717329 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:04:08.739920 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.739890 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:04:08.840377 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.840322 2559 generic.go:356] "Generic (PLEG): container finished" podID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerID="3796abaee6215aa6a1623a86ff921bf835081c0716345123b885d4a4af566a31" exitCode=2 Sep 16 19:04:08.840377 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.840353 2559 generic.go:356] "Generic (PLEG): container finished" podID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerID="2ed410c8eed207039b46884fbafb0bf643a2eb46db6ff39456a9c2c4be97f5a0" exitCode=2 Sep 16 19:04:08.840377 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.840352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" event={"ID":"b2e9d1eb-3932-44a0-9bbb-f023649c868d","Type":"ContainerDied","Data":"3796abaee6215aa6a1623a86ff921bf835081c0716345123b885d4a4af566a31"} Sep 16 19:04:08.840790 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.840423 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" event={"ID":"b2e9d1eb-3932-44a0-9bbb-f023649c868d","Type":"ContainerDied","Data":"2ed410c8eed207039b46884fbafb0bf643a2eb46db6ff39456a9c2c4be97f5a0"} Sep 16 19:04:08.843890 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.843862 2559 generic.go:356] "Generic (PLEG): container finished" podID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerID="03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078" exitCode=2 Sep 16 19:04:08.844052 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.843954 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerDied","Data":"03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078"} Sep 16 19:04:08.848950 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.848926 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:04:08.900090 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.900068 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:04:08.981787 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.981698 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lxllz\" (UniqueName: \"kubernetes.io/projected/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-api-access-lxllz\") pod \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " Sep 16 19:04:08.981787 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.981739 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls\") pod \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " Sep 16 19:04:08.981787 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.981763 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-rbac-proxy-sar-config\") pod \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\" (UID: \"b2e9d1eb-3932-44a0-9bbb-f023649c868d\") " Sep 16 19:04:08.982220 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.982192 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2e9d1eb-3932-44a0-9bbb-f023649c868d" (UID: "b2e9d1eb-3932-44a0-9bbb-f023649c868d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:04:08.983886 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.983864 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-api-access-lxllz" (OuterVolumeSpecName: "kube-api-access-lxllz") pod "b2e9d1eb-3932-44a0-9bbb-f023649c868d" (UID: "b2e9d1eb-3932-44a0-9bbb-f023649c868d"). InnerVolumeSpecName "kube-api-access-lxllz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:04:08.983886 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:08.983869 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2e9d1eb-3932-44a0-9bbb-f023649c868d" (UID: "b2e9d1eb-3932-44a0-9bbb-f023649c868d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:04:09.082879 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.082826 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxllz\" (UniqueName: \"kubernetes.io/projected/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-api-access-lxllz\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:04:09.082879 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.082870 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2e9d1eb-3932-44a0-9bbb-f023649c868d-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:04:09.082879 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.082880 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2e9d1eb-3932-44a0-9bbb-f023649c868d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:04:09.848907 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.848874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" event={"ID":"b2e9d1eb-3932-44a0-9bbb-f023649c868d","Type":"ContainerDied","Data":"175cbf604f726e9aeb70fb2dfc9d60ee06649ad859651c3eb14bb7e1dab72df4"} Sep 16 19:04:09.849361 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.848951 2559 scope.go:117] "RemoveContainer" containerID="3796abaee6215aa6a1623a86ff921bf835081c0716345123b885d4a4af566a31" Sep 16 19:04:09.849361 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.848883 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5" Sep 16 19:04:09.857016 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.856995 2559 scope.go:117] "RemoveContainer" containerID="2ed410c8eed207039b46884fbafb0bf643a2eb46db6ff39456a9c2c4be97f5a0" Sep 16 19:04:09.870132 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.870084 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5"] Sep 16 19:04:09.874073 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:09.874053 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-2d3bd-predictor-84cbc66dbf-mdbj5"] Sep 16 19:04:10.721170 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:10.721128 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" path="/var/lib/kubelet/pods/b2e9d1eb-3932-44a0-9bbb-f023649c868d/volumes" Sep 16 19:04:10.875708 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:10.875675 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:04:11.511260 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:11.511225 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused" Sep 16 19:04:13.867262 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:13.867226 2559 generic.go:356] "Generic (PLEG): container finished" podID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerID="257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71" exitCode=0 Sep 16 19:04:13.867616 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:13.867296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerDied","Data":"257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71"} Sep 16 19:04:16.511341 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:16.511297 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused" Sep 16 19:04:18.716921 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:18.716877 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:04:18.717359 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:18.717199 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:04:21.511582 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:21.511539 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused" Sep 16 19:04:21.511999 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:21.511658 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:04:26.511185 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:26.511144 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused" Sep 16 19:04:28.717159 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:28.717119 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused" Sep 16 19:04:28.717575 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:28.717550 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:04:28.720449 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:28.720432 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:04:28.720534 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:28.720522 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:04:31.511414 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:31.511357 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused" Sep 16 19:04:36.511106 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:36.511024 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused" Sep 16 19:04:38.749124 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.749100 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:04:38.896036 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.895951 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lts59\" (UniqueName: \"kubernetes.io/projected/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-api-access-lts59\") pod \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " Sep 16 19:04:38.896036 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.896015 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-rbac-proxy-sar-config\") pod \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " Sep 16 19:04:38.896212 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.896057 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10933a9d-e7fd-4b1a-9eda-a21f48f74639-proxy-tls\") pod \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " Sep 16 19:04:38.896260 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.896238 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-cabundle-cert\") pod \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " Sep 16 19:04:38.896326 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.896310 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kserve-provision-location\") pod \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\" (UID: \"10933a9d-e7fd-4b1a-9eda-a21f48f74639\") " Sep 16 19:04:38.896431 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.896307 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "10933a9d-e7fd-4b1a-9eda-a21f48f74639" (UID: "10933a9d-e7fd-4b1a-9eda-a21f48f74639"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:04:38.896562 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.896498 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "10933a9d-e7fd-4b1a-9eda-a21f48f74639" (UID: "10933a9d-e7fd-4b1a-9eda-a21f48f74639"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:04:38.896562 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.896536 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:04:38.896668 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.896557 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "10933a9d-e7fd-4b1a-9eda-a21f48f74639" (UID: "10933a9d-e7fd-4b1a-9eda-a21f48f74639"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:04:38.897877 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.897855 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-api-access-lts59" (OuterVolumeSpecName: "kube-api-access-lts59") pod "10933a9d-e7fd-4b1a-9eda-a21f48f74639" (UID: "10933a9d-e7fd-4b1a-9eda-a21f48f74639"). InnerVolumeSpecName "kube-api-access-lts59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:04:38.897982 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.897942 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10933a9d-e7fd-4b1a-9eda-a21f48f74639-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "10933a9d-e7fd-4b1a-9eda-a21f48f74639" (UID: "10933a9d-e7fd-4b1a-9eda-a21f48f74639"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:04:38.959903 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.959872 2559 generic.go:356] "Generic (PLEG): container finished" podID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerID="8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb" exitCode=0 Sep 16 19:04:38.960054 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.959916 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerDied","Data":"8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb"} Sep 16 19:04:38.960054 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.959936 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" Sep 16 19:04:38.960054 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.959991 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" event={"ID":"10933a9d-e7fd-4b1a-9eda-a21f48f74639","Type":"ContainerDied","Data":"81c21f0ce82786fb778e4e508267f9f7bbb31a5c9da2ec69589524b02fff62a2"} Sep 16 19:04:38.960054 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.960030 2559 scope.go:117] "RemoveContainer" containerID="8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb" Sep 16 19:04:38.968076 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.968057 2559 scope.go:117] "RemoveContainer" containerID="03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078" Sep 16 19:04:38.979856 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.979836 2559 scope.go:117] "RemoveContainer" containerID="257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71" Sep 16 19:04:38.986305 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.986290 2559 scope.go:117] "RemoveContainer" containerID="b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322" Sep 16 19:04:38.992530 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.992513 2559 scope.go:117] "RemoveContainer" containerID="8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb" Sep 16 19:04:38.992802 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:04:38.992782 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb\": container with ID starting with 8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb not found: ID does not exist" containerID="8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb" Sep 16 19:04:38.992864 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.992810 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb"} err="failed to get container status \"8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb\": rpc error: code = NotFound desc = could not find container \"8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb\": container with ID starting with 8e287f15fb95b767827f3d3a428545e6cb281de8f539e152be2482d10031aecb not found: ID does not exist" Sep 16 19:04:38.992864 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.992825 2559 scope.go:117] "RemoveContainer" containerID="03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078" Sep 16 19:04:38.993127 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:04:38.993094 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078\": container with ID starting with 03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078 not found: ID does not exist" containerID="03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078" Sep 16 19:04:38.993202 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.993136 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078"} err="failed to get container status \"03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078\": rpc error: code = NotFound desc = could not find container \"03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078\": container with ID starting with 03a451b1fc82e599ea7770746a15b78dfcce5e2cdb3aad5286705a44ebde5078 not found: ID does not exist" Sep 16 19:04:38.993202 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.993158 2559 scope.go:117] "RemoveContainer" containerID="257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71" Sep 16 19:04:38.993493 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:04:38.993465 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71\": container with ID starting with 257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71 not found: ID does not exist" containerID="257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71" Sep 16 19:04:38.993584 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.993499 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71"} err="failed to get container status \"257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71\": rpc error: code = NotFound desc = could not find container \"257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71\": container with ID starting with 257efc5a72bec5dc1529231119524db1b077e19b7b8dd2a1276765a27c9dae71 not found: ID does not exist" Sep 16 19:04:38.993584 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.993519 2559 scope.go:117] "RemoveContainer" containerID="b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322" Sep 16 19:04:38.993862 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:04:38.993836 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322\": container with ID starting with b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322 not found: ID does not exist" containerID="b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322" Sep 16 19:04:38.993950 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.993887 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322"} err="failed to get container status \"b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322\": rpc error: code = NotFound desc = could not find container \"b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322\": container with ID starting with b9f8c6a9cedb864c568d36c399d2b54094708e46c19a12046f6a49619e988322 not found: ID does not exist" Sep 16 19:04:38.994654 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.994637 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh"] Sep 16 19:04:38.996919 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.996900 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/10933a9d-e7fd-4b1a-9eda-a21f48f74639-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:04:38.996919 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.996919 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/10933a9d-e7fd-4b1a-9eda-a21f48f74639-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:04:38.997073 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.996928 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:04:38.997073 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.996937 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lts59\" (UniqueName: \"kubernetes.io/projected/10933a9d-e7fd-4b1a-9eda-a21f48f74639-kube-api-access-lts59\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:04:38.998624 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:38.998605 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh"] Sep 16 19:04:39.716958 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:39.716915 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" probeResult="failure" output="Get \"http://10.132.0.23:9081/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 16 19:04:39.717216 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:39.716917 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-2d3bd-predictor-7fdc485bcd-mhsdh" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: i/o timeout" Sep 16 19:04:40.721361 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:40.721328 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" path="/var/lib/kubelet/pods/10933a9d-e7fd-4b1a-9eda-a21f48f74639/volumes" Sep 16 19:04:52.679101 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:04:52.679061 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:12.658807 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:12.658776 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 19:05:12.660725 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:12.660705 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 19:05:32.043980 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:32.043946 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:37.180812 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.180771 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:37.203199 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203166 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx"] Sep 16 19:05:37.203432 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203418 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" Sep 16 19:05:37.203474 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203436 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" Sep 16 19:05:37.203474 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203447 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerName="kube-rbac-proxy" Sep 16 19:05:37.203474 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203453 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerName="kube-rbac-proxy" Sep 16 19:05:37.203474 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203460 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerName="kserve-container" Sep 16 19:05:37.203474 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203466 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerName="kserve-container" Sep 16 19:05:37.203474 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203474 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203479 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203488 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203493 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203502 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203509 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203519 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kube-rbac-proxy" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203524 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kube-rbac-proxy" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203530 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="storage-initializer" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203535 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="storage-initializer" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203540 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="storage-initializer" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203545 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="storage-initializer" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203587 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerName="kserve-container" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203597 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kserve-container" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203603 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="agent" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203609 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2e9d1eb-3932-44a0-9bbb-f023649c868d" containerName="kube-rbac-proxy" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203615 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="10933a9d-e7fd-4b1a-9eda-a21f48f74639" containerName="kube-rbac-proxy" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203622 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kserve-container" Sep 16 19:05:37.203648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.203627 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cbfecba-d5da-4c61-aa73-d780c62d87d1" containerName="kube-rbac-proxy" Sep 16 19:05:37.207854 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.207831 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.211196 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.211171 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-fcf1d-predictor-serving-cert\"" Sep 16 19:05:37.211368 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.211208 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 19:05:37.211442 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.211250 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 19:05:37.211501 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.211456 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\"" Sep 16 19:05:37.212582 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.212563 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:05:37.212643 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.212621 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-fcf1d-kube-rbac-proxy-sar-config\"" Sep 16 19:05:37.212643 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.212636 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 19:05:37.221620 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.221597 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx"] Sep 16 19:05:37.297809 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.297772 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-cabundle-cert\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.297809 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.297809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.298065 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.297831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.298065 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.297916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380f3970-3d65-4430-bcdb-35aba93ce880-kserve-provision-location\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.298065 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.298004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-569mv\" (UniqueName: \"kubernetes.io/projected/380f3970-3d65-4430-bcdb-35aba93ce880-kube-api-access-569mv\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.398316 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.398284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380f3970-3d65-4430-bcdb-35aba93ce880-kserve-provision-location\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.398482 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.398327 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-569mv\" (UniqueName: \"kubernetes.io/projected/380f3970-3d65-4430-bcdb-35aba93ce880-kube-api-access-569mv\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.398482 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.398354 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-cabundle-cert\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.398482 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.398381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.398482 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.398403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.398696 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:05:37.398536 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-serving-cert: secret "raw-sklearn-runtime-fcf1d-predictor-serving-cert" not found Sep 16 19:05:37.398696 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:05:37.398616 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls podName:380f3970-3d65-4430-bcdb-35aba93ce880 nodeName:}" failed. No retries permitted until 2026-09-16 19:05:37.898596677 +0000 UTC m=+1225.697718495 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls") pod "raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" (UID: "380f3970-3d65-4430-bcdb-35aba93ce880") : secret "raw-sklearn-runtime-fcf1d-predictor-serving-cert" not found Sep 16 19:05:37.398798 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.398743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380f3970-3d65-4430-bcdb-35aba93ce880-kserve-provision-location\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.399176 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.399155 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-cabundle-cert\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.399224 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.399155 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.408186 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.408169 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-569mv\" (UniqueName: \"kubernetes.io/projected/380f3970-3d65-4430-bcdb-35aba93ce880-kube-api-access-569mv\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.762018 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.761988 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:05:37.902958 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.902925 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:37.905324 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:37.905295 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls\") pod \"raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:38.118389 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:38.118309 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:38.396732 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:38.396651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx"] Sep 16 19:05:38.400694 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:05:38.400667 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380f3970_3d65_4430_bcdb_35aba93ce880.slice/crio-7c84cc19c0dc1562cb81285fe326e69762e1f3c01142498b42f01efa57e3a77f WatchSource:0}: Error finding container 7c84cc19c0dc1562cb81285fe326e69762e1f3c01142498b42f01efa57e3a77f: Status 404 returned error can't find the container with id 7c84cc19c0dc1562cb81285fe326e69762e1f3c01142498b42f01efa57e3a77f Sep 16 19:05:39.162275 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:39.162240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" event={"ID":"380f3970-3d65-4430-bcdb-35aba93ce880","Type":"ContainerStarted","Data":"36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25"} Sep 16 19:05:39.162442 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:39.162311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" event={"ID":"380f3970-3d65-4430-bcdb-35aba93ce880","Type":"ContainerStarted","Data":"7c84cc19c0dc1562cb81285fe326e69762e1f3c01142498b42f01efa57e3a77f"} Sep 16 19:05:42.174130 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:42.174102 2559 generic.go:356] "Generic (PLEG): container finished" podID="380f3970-3d65-4430-bcdb-35aba93ce880" containerID="36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25" exitCode=0 Sep 16 19:05:42.174519 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:42.174164 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" event={"ID":"380f3970-3d65-4430-bcdb-35aba93ce880","Type":"ContainerDied","Data":"36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25"} Sep 16 19:05:43.179530 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:43.179491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" event={"ID":"380f3970-3d65-4430-bcdb-35aba93ce880","Type":"ContainerStarted","Data":"8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af"} Sep 16 19:05:43.180019 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:43.179565 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" event={"ID":"380f3970-3d65-4430-bcdb-35aba93ce880","Type":"ContainerStarted","Data":"6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab"} Sep 16 19:05:43.180019 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:43.179783 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:43.203570 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:43.203517 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podStartSLOduration=2.517435375 podStartE2EDuration="6.203502552s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.686067177s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:05:43.202304892 +0000 UTC m=+1231.001426752" watchObservedRunningTime="2026-09-16 19:05:43.203502552 +0000 UTC m=+1231.002624391" Sep 16 19:05:44.183423 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:44.183393 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:44.184635 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:44.184608 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:05:45.186943 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:45.186899 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:05:50.191331 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:50.191298 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:05:50.191887 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:05:50.191860 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:06:00.192165 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:06:00.192127 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:06:10.192699 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:06:10.192659 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:06:20.192787 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:06:20.192747 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:06:30.192073 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:06:30.192037 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:06:40.192353 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:06:40.192311 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:06:50.192479 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:06:50.192436 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:06:57.717734 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:06:57.717699 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:06:59.054649 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:06:59.054608 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:04.100519 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.100484 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:04.148211 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.148178 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947"] Sep 16 19:07:04.151549 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.151528 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.156760 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.156736 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\"" Sep 16 19:07:04.157063 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.157042 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Sep 16 19:07:04.182756 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.182729 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947"] Sep 16 19:07:04.238474 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.238443 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg8jk\" (UniqueName: \"kubernetes.io/projected/50f20665-6cb7-4b95-b94f-a649485c5050-kube-api-access-rg8jk\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.238635 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.238485 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f20665-6cb7-4b95-b94f-a649485c5050-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.238635 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.238540 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.238635 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.238584 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.238635 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.238605 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f20665-6cb7-4b95-b94f-a649485c5050-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.239925 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.239903 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:04.340023 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.339986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg8jk\" (UniqueName: \"kubernetes.io/projected/50f20665-6cb7-4b95-b94f-a649485c5050-kube-api-access-rg8jk\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.340023 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.340051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f20665-6cb7-4b95-b94f-a649485c5050-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.340316 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.340087 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.340316 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.340290 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.340429 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.340321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f20665-6cb7-4b95-b94f-a649485c5050-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.340574 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.340517 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f20665-6cb7-4b95-b94f-a649485c5050-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.341002 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.340954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.341002 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.340991 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.342999 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.342980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f20665-6cb7-4b95-b94f-a649485c5050-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.348787 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.348768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg8jk\" (UniqueName: \"kubernetes.io/projected/50f20665-6cb7-4b95-b94f-a649485c5050-kube-api-access-rg8jk\") pod \"isvc-canary-create-predictor-6fb974599-5t947\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.369004 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.368926 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:04.461891 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.461865 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:04.737458 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:04.737360 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947"] Sep 16 19:07:04.740011 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:07:04.739983 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f20665_6cb7_4b95_b94f_a649485c5050.slice/crio-7069b08fdce67fe3603fdd74d15bde90fab309dbe0c16f25298074ddbeaf539b WatchSource:0}: Error finding container 7069b08fdce67fe3603fdd74d15bde90fab309dbe0c16f25298074ddbeaf539b: Status 404 returned error can't find the container with id 7069b08fdce67fe3603fdd74d15bde90fab309dbe0c16f25298074ddbeaf539b Sep 16 19:07:05.465856 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:05.465825 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" event={"ID":"50f20665-6cb7-4b95-b94f-a649485c5050","Type":"ContainerStarted","Data":"559165becf92782ab3189108f38890841d080e464897f70fa5e003e9cee36530"} Sep 16 19:07:05.466262 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:05.465877 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" event={"ID":"50f20665-6cb7-4b95-b94f-a649485c5050","Type":"ContainerStarted","Data":"7069b08fdce67fe3603fdd74d15bde90fab309dbe0c16f25298074ddbeaf539b"} Sep 16 19:07:07.264645 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:07.264613 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx"] Sep 16 19:07:07.265027 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:07.264846 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" containerID="cri-o://6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab" gracePeriod=30 Sep 16 19:07:07.265027 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:07.264896 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kube-rbac-proxy" containerID="cri-o://8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af" gracePeriod=30 Sep 16 19:07:07.299468 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:07.299444 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:07:07.480349 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:07.480317 2559 generic.go:356] "Generic (PLEG): container finished" podID="380f3970-3d65-4430-bcdb-35aba93ce880" containerID="8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af" exitCode=2 Sep 16 19:07:07.480500 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:07.480390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" event={"ID":"380f3970-3d65-4430-bcdb-35aba93ce880","Type":"ContainerDied","Data":"8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af"} Sep 16 19:07:07.717904 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:07.717865 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused" Sep 16 19:07:08.484669 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:08.484636 2559 generic.go:356] "Generic (PLEG): container finished" podID="50f20665-6cb7-4b95-b94f-a649485c5050" containerID="559165becf92782ab3189108f38890841d080e464897f70fa5e003e9cee36530" exitCode=0 Sep 16 19:07:08.485073 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:08.484709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" event={"ID":"50f20665-6cb7-4b95-b94f-a649485c5050","Type":"ContainerDied","Data":"559165becf92782ab3189108f38890841d080e464897f70fa5e003e9cee36530"} Sep 16 19:07:09.490853 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:09.490821 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" event={"ID":"50f20665-6cb7-4b95-b94f-a649485c5050","Type":"ContainerStarted","Data":"d79e6b2f197d83aafa54893b912b035ffaa992431041e2441b5cc310598a772d"} Sep 16 19:07:09.491264 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:09.490878 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" event={"ID":"50f20665-6cb7-4b95-b94f-a649485c5050","Type":"ContainerStarted","Data":"4be8bac7643063c224ec1580944dffce959b121521f493718e82b2ea17919da7"} Sep 16 19:07:09.491264 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:09.491080 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:09.513083 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:09.513036 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podStartSLOduration=1.907920726 podStartE2EDuration="5.513021756s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.60510103s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:09.51151733 +0000 UTC m=+1317.310639181" watchObservedRunningTime="2026-09-16 19:07:09.513021756 +0000 UTC m=+1317.312143598" Sep 16 19:07:10.187465 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:10.187421 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused" Sep 16 19:07:10.495185 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:10.495101 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:10.496336 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:10.496307 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:07:11.498137 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.498097 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:07:11.902688 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.902664 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:07:11.990487 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.990462 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-cabundle-cert\") pod \"380f3970-3d65-4430-bcdb-35aba93ce880\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " Sep 16 19:07:11.990647 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.990496 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-kube-rbac-proxy-sar-config\") pod \"380f3970-3d65-4430-bcdb-35aba93ce880\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " Sep 16 19:07:11.990647 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.990516 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls\") pod \"380f3970-3d65-4430-bcdb-35aba93ce880\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " Sep 16 19:07:11.990647 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.990548 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-569mv\" (UniqueName: \"kubernetes.io/projected/380f3970-3d65-4430-bcdb-35aba93ce880-kube-api-access-569mv\") pod \"380f3970-3d65-4430-bcdb-35aba93ce880\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " Sep 16 19:07:11.990778 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.990676 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380f3970-3d65-4430-bcdb-35aba93ce880-kserve-provision-location\") pod \"380f3970-3d65-4430-bcdb-35aba93ce880\" (UID: \"380f3970-3d65-4430-bcdb-35aba93ce880\") " Sep 16 19:07:11.990956 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.990928 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "380f3970-3d65-4430-bcdb-35aba93ce880" (UID: "380f3970-3d65-4430-bcdb-35aba93ce880"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:11.991099 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.990953 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "380f3970-3d65-4430-bcdb-35aba93ce880" (UID: "380f3970-3d65-4430-bcdb-35aba93ce880"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:07:11.991099 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.991040 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380f3970-3d65-4430-bcdb-35aba93ce880-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "380f3970-3d65-4430-bcdb-35aba93ce880" (UID: "380f3970-3d65-4430-bcdb-35aba93ce880"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:07:11.992618 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.992594 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380f3970-3d65-4430-bcdb-35aba93ce880-kube-api-access-569mv" (OuterVolumeSpecName: "kube-api-access-569mv") pod "380f3970-3d65-4430-bcdb-35aba93ce880" (UID: "380f3970-3d65-4430-bcdb-35aba93ce880"). InnerVolumeSpecName "kube-api-access-569mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:07:11.992618 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:11.992605 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "380f3970-3d65-4430-bcdb-35aba93ce880" (UID: "380f3970-3d65-4430-bcdb-35aba93ce880"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:07:12.091380 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.091286 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:07:12.091380 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.091325 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/380f3970-3d65-4430-bcdb-35aba93ce880-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:07:12.091380 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.091337 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/380f3970-3d65-4430-bcdb-35aba93ce880-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:07:12.091380 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.091346 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-569mv\" (UniqueName: \"kubernetes.io/projected/380f3970-3d65-4430-bcdb-35aba93ce880-kube-api-access-569mv\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:07:12.091380 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.091355 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/380f3970-3d65-4430-bcdb-35aba93ce880-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:07:12.503273 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.503241 2559 generic.go:356] "Generic (PLEG): container finished" podID="380f3970-3d65-4430-bcdb-35aba93ce880" containerID="6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab" exitCode=0 Sep 16 19:07:12.503647 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.503299 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" Sep 16 19:07:12.503647 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.503306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" event={"ID":"380f3970-3d65-4430-bcdb-35aba93ce880","Type":"ContainerDied","Data":"6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab"} Sep 16 19:07:12.503647 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.503352 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx" event={"ID":"380f3970-3d65-4430-bcdb-35aba93ce880","Type":"ContainerDied","Data":"7c84cc19c0dc1562cb81285fe326e69762e1f3c01142498b42f01efa57e3a77f"} Sep 16 19:07:12.503647 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.503377 2559 scope.go:117] "RemoveContainer" containerID="8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af" Sep 16 19:07:12.511618 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.511595 2559 scope.go:117] "RemoveContainer" containerID="6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab" Sep 16 19:07:12.519388 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.519367 2559 scope.go:117] "RemoveContainer" containerID="36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25" Sep 16 19:07:12.527139 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.527124 2559 scope.go:117] "RemoveContainer" containerID="8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af" Sep 16 19:07:12.527446 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:07:12.527418 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af\": container with ID starting with 8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af not found: ID does not exist" containerID="8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af" Sep 16 19:07:12.527525 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.527466 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af"} err="failed to get container status \"8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af\": rpc error: code = NotFound desc = could not find container \"8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af\": container with ID starting with 8ed0e5f9e3078542f84a1643433276e899efdf4a91f68a764ed3f18fdfb278af not found: ID does not exist" Sep 16 19:07:12.527525 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.527484 2559 scope.go:117] "RemoveContainer" containerID="6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab" Sep 16 19:07:12.527643 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.527527 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx"] Sep 16 19:07:12.527771 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:07:12.527754 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab\": container with ID starting with 6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab not found: ID does not exist" containerID="6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab" Sep 16 19:07:12.527982 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.527773 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab"} err="failed to get container status \"6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab\": rpc error: code = NotFound desc = could not find container \"6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab\": container with ID starting with 6bb77bd2f278d1119c749e547db2dee2a58fd14b1c2f275101a1f1b83154f4ab not found: ID does not exist" Sep 16 19:07:12.527982 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.527789 2559 scope.go:117] "RemoveContainer" containerID="36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25" Sep 16 19:07:12.528173 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:07:12.528107 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25\": container with ID starting with 36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25 not found: ID does not exist" containerID="36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25" Sep 16 19:07:12.528173 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.528138 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25"} err="failed to get container status \"36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25\": rpc error: code = NotFound desc = could not find container \"36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25\": container with ID starting with 36c83c0a272bbd9af8480690385e371265c4389ec83c91a3de7380ff7ba9cc25 not found: ID does not exist" Sep 16 19:07:12.531203 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.531185 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-fcf1d-predictor-558b545566-26mdx"] Sep 16 19:07:12.721284 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:12.721253 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" path="/var/lib/kubelet/pods/380f3970-3d65-4430-bcdb-35aba93ce880/volumes" Sep 16 19:07:16.502770 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:16.502742 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:07:16.503329 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:16.503301 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:07:26.503382 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:26.503343 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:07:36.503724 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:36.503635 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:07:46.504105 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:46.504065 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:07:56.503918 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:07:56.503880 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:08:06.503777 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:06.503738 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:08:16.503543 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:16.503502 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:08:23.772562 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:23.772525 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:25.527700 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:25.527668 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:26.503802 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:26.503773 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:08:34.182167 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:34.182135 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947"] Sep 16 19:08:34.182645 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:34.182338 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" containerID="cri-o://4be8bac7643063c224ec1580944dffce959b121521f493718e82b2ea17919da7" gracePeriod=30 Sep 16 19:08:34.182645 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:34.182379 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kube-rbac-proxy" containerID="cri-o://d79e6b2f197d83aafa54893b912b035ffaa992431041e2441b5cc310598a772d" gracePeriod=30 Sep 16 19:08:34.251209 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:34.251173 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:34.359822 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:34.359792 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:34.798088 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:34.798054 2559 generic.go:356] "Generic (PLEG): container finished" podID="50f20665-6cb7-4b95-b94f-a649485c5050" containerID="d79e6b2f197d83aafa54893b912b035ffaa992431041e2441b5cc310598a772d" exitCode=2 Sep 16 19:08:34.798240 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:34.798110 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" event={"ID":"50f20665-6cb7-4b95-b94f-a649485c5050","Type":"ContainerDied","Data":"d79e6b2f197d83aafa54893b912b035ffaa992431041e2441b5cc310598a772d"} Sep 16 19:08:34.954531 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:34.954499 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:35.054550 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:35.054461 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:35.142425 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:35.142391 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:36.498627 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:36.498587 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused" Sep 16 19:08:36.504091 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:36.504062 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused" Sep 16 19:08:38.814707 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.814678 2559 generic.go:356] "Generic (PLEG): container finished" podID="50f20665-6cb7-4b95-b94f-a649485c5050" containerID="4be8bac7643063c224ec1580944dffce959b121521f493718e82b2ea17919da7" exitCode=0 Sep 16 19:08:38.815078 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.814736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" event={"ID":"50f20665-6cb7-4b95-b94f-a649485c5050","Type":"ContainerDied","Data":"4be8bac7643063c224ec1580944dffce959b121521f493718e82b2ea17919da7"} Sep 16 19:08:38.815078 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.814797 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" event={"ID":"50f20665-6cb7-4b95-b94f-a649485c5050","Type":"ContainerDied","Data":"7069b08fdce67fe3603fdd74d15bde90fab309dbe0c16f25298074ddbeaf539b"} Sep 16 19:08:38.815078 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.814831 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7069b08fdce67fe3603fdd74d15bde90fab309dbe0c16f25298074ddbeaf539b" Sep 16 19:08:38.821761 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.821741 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:08:38.919201 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-kube-rbac-proxy-sar-config\") pod \"50f20665-6cb7-4b95-b94f-a649485c5050\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " Sep 16 19:08:38.919353 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919219 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg8jk\" (UniqueName: \"kubernetes.io/projected/50f20665-6cb7-4b95-b94f-a649485c5050-kube-api-access-rg8jk\") pod \"50f20665-6cb7-4b95-b94f-a649485c5050\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " Sep 16 19:08:38.919353 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919259 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f20665-6cb7-4b95-b94f-a649485c5050-proxy-tls\") pod \"50f20665-6cb7-4b95-b94f-a649485c5050\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " Sep 16 19:08:38.919475 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919366 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-cabundle-cert\") pod \"50f20665-6cb7-4b95-b94f-a649485c5050\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " Sep 16 19:08:38.919475 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919429 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f20665-6cb7-4b95-b94f-a649485c5050-kserve-provision-location\") pod \"50f20665-6cb7-4b95-b94f-a649485c5050\" (UID: \"50f20665-6cb7-4b95-b94f-a649485c5050\") " Sep 16 19:08:38.919571 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919546 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "50f20665-6cb7-4b95-b94f-a649485c5050" (UID: "50f20665-6cb7-4b95-b94f-a649485c5050"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:08:38.919653 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919637 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:08:38.919739 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919711 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f20665-6cb7-4b95-b94f-a649485c5050-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "50f20665-6cb7-4b95-b94f-a649485c5050" (UID: "50f20665-6cb7-4b95-b94f-a649485c5050"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:08:38.919846 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.919749 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "50f20665-6cb7-4b95-b94f-a649485c5050" (UID: "50f20665-6cb7-4b95-b94f-a649485c5050"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:08:38.921150 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.921122 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f20665-6cb7-4b95-b94f-a649485c5050-kube-api-access-rg8jk" (OuterVolumeSpecName: "kube-api-access-rg8jk") pod "50f20665-6cb7-4b95-b94f-a649485c5050" (UID: "50f20665-6cb7-4b95-b94f-a649485c5050"). InnerVolumeSpecName "kube-api-access-rg8jk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:08:38.921229 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:38.921183 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50f20665-6cb7-4b95-b94f-a649485c5050-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "50f20665-6cb7-4b95-b94f-a649485c5050" (UID: "50f20665-6cb7-4b95-b94f-a649485c5050"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:08:39.019950 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.019919 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/50f20665-6cb7-4b95-b94f-a649485c5050-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:08:39.019950 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.019943 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/50f20665-6cb7-4b95-b94f-a649485c5050-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:08:39.019950 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.019953 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/50f20665-6cb7-4b95-b94f-a649485c5050-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:08:39.020209 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.019976 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg8jk\" (UniqueName: \"kubernetes.io/projected/50f20665-6cb7-4b95-b94f-a649485c5050-kube-api-access-rg8jk\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:08:39.296327 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.296249 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:39.341184 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341152 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk"] Sep 16 19:08:39.341395 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341384 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kube-rbac-proxy" Sep 16 19:08:39.341438 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341397 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kube-rbac-proxy" Sep 16 19:08:39.341438 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341410 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" Sep 16 19:08:39.341438 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341416 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" Sep 16 19:08:39.341438 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341427 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" Sep 16 19:08:39.341438 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341433 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341440 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kube-rbac-proxy" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341446 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kube-rbac-proxy" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341452 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="storage-initializer" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341457 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="storage-initializer" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341464 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="storage-initializer" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341469 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="storage-initializer" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341508 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kserve-container" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341516 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kserve-container" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341522 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" containerName="kube-rbac-proxy" Sep 16 19:08:39.341593 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.341529 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="380f3970-3d65-4430-bcdb-35aba93ce880" containerName="kube-rbac-proxy" Sep 16 19:08:39.344670 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.344649 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.348453 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.348434 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Sep 16 19:08:39.348528 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.348478 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Sep 16 19:08:39.360064 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.360042 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk"] Sep 16 19:08:39.408696 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.408663 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:39.417639 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.417612 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf"] Sep 16 19:08:39.420804 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.420788 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.425499 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.424925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.425499 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.425009 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/164271d6-2621-4e8c-a224-3c1e44dc24c7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.425499 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.425064 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54ppf\" (UniqueName: \"kubernetes.io/projected/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-api-access-54ppf\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.425499 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.425120 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.425499 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.425161 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.425499 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.425362 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\"" Sep 16 19:08:39.438260 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.438235 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf"] Sep 16 19:08:39.526423 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.526423 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/164271d6-2621-4e8c-a224-3c1e44dc24c7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.526646 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526445 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.526646 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ppf\" (UniqueName: \"kubernetes.io/projected/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-api-access-54ppf\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.526646 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526509 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.526646 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526530 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6pmj\" (UniqueName: \"kubernetes.io/projected/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-api-access-p6pmj\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.526646 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.526646 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526578 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.526646 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.526947 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:08:39.526673 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-serving-cert: secret "isvc-canary-promote-v2-predictor-serving-cert" not found Sep 16 19:08:39.526947 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:08:39.526734 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls podName:164271d6-2621-4e8c-a224-3c1e44dc24c7 nodeName:}" failed. No retries permitted until 2026-09-16 19:08:40.026717441 +0000 UTC m=+1407.825839257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls") pod "isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" (UID: "164271d6-2621-4e8c-a224-3c1e44dc24c7") : secret "isvc-canary-promote-v2-predictor-serving-cert" not found Sep 16 19:08:39.526947 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.526947 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.526842 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/164271d6-2621-4e8c-a224-3c1e44dc24c7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.527144 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.527127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.527181 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.527146 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.536171 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.536152 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ppf\" (UniqueName: \"kubernetes.io/projected/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-api-access-54ppf\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:39.628081 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.628008 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.628081 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.628060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6pmj\" (UniqueName: \"kubernetes.io/projected/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-api-access-p6pmj\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.628283 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.628100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.628283 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.628119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.628283 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.628142 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.628283 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:08:39.628261 2559 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found Sep 16 19:08:39.628480 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:08:39.628338 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls podName:9562e8a4-b4f6-4ec1-8145-9403ca36f85d nodeName:}" failed. No retries permitted until 2026-09-16 19:08:40.128318738 +0000 UTC m=+1407.927440561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls") pod "isvc-canary-promote-predictor-649dd98487-2lzlf" (UID: "9562e8a4-b4f6-4ec1-8145-9403ca36f85d") : secret "isvc-canary-promote-predictor-serving-cert" not found Sep 16 19:08:39.628542 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.628489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.628712 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.628694 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.628749 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.628698 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.637289 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.637264 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6pmj\" (UniqueName: \"kubernetes.io/projected/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-api-access-p6pmj\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:39.817631 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.817602 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947" Sep 16 19:08:39.841296 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.841269 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947"] Sep 16 19:08:39.847112 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.847092 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-5t947"] Sep 16 19:08:39.893144 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:39.893087 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:40.011690 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.011662 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:08:40.030869 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.030843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:40.033325 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.033305 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:40.131599 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.131547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:40.133914 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.133889 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-2lzlf\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:40.254590 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.254560 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:40.332645 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.332614 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:40.526720 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.526699 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk"] Sep 16 19:08:40.528594 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:08:40.528567 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164271d6_2621_4e8c_a224_3c1e44dc24c7.slice/crio-7dc58ea9d349cb694f7ae909e04a540d371baeb3a149b2529fde39a8a77adb2d WatchSource:0}: Error finding container 7dc58ea9d349cb694f7ae909e04a540d371baeb3a149b2529fde39a8a77adb2d: Status 404 returned error can't find the container with id 7dc58ea9d349cb694f7ae909e04a540d371baeb3a149b2529fde39a8a77adb2d Sep 16 19:08:40.606838 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.606810 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf"] Sep 16 19:08:40.609498 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:08:40.609473 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9562e8a4_b4f6_4ec1_8145_9403ca36f85d.slice/crio-2818f7ec712d94c927a115323401427882f3233136802c0b6e7311bbfc23565d WatchSource:0}: Error finding container 2818f7ec712d94c927a115323401427882f3233136802c0b6e7311bbfc23565d: Status 404 returned error can't find the container with id 2818f7ec712d94c927a115323401427882f3233136802c0b6e7311bbfc23565d Sep 16 19:08:40.721705 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.721661 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50f20665-6cb7-4b95-b94f-a649485c5050" path="/var/lib/kubelet/pods/50f20665-6cb7-4b95-b94f-a649485c5050/volumes" Sep 16 19:08:40.823589 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.823458 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" event={"ID":"164271d6-2621-4e8c-a224-3c1e44dc24c7","Type":"ContainerStarted","Data":"aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40"} Sep 16 19:08:40.823589 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.823529 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" event={"ID":"164271d6-2621-4e8c-a224-3c1e44dc24c7","Type":"ContainerStarted","Data":"7dc58ea9d349cb694f7ae909e04a540d371baeb3a149b2529fde39a8a77adb2d"} Sep 16 19:08:40.825297 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.825267 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" event={"ID":"9562e8a4-b4f6-4ec1-8145-9403ca36f85d","Type":"ContainerStarted","Data":"38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545"} Sep 16 19:08:40.825413 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:40.825318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" event={"ID":"9562e8a4-b4f6-4ec1-8145-9403ca36f85d","Type":"ContainerStarted","Data":"2818f7ec712d94c927a115323401427882f3233136802c0b6e7311bbfc23565d"} Sep 16 19:08:44.840543 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:44.840456 2559 generic.go:356] "Generic (PLEG): container finished" podID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerID="aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40" exitCode=0 Sep 16 19:08:44.840980 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:44.840539 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" event={"ID":"164271d6-2621-4e8c-a224-3c1e44dc24c7","Type":"ContainerDied","Data":"aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40"} Sep 16 19:08:44.842192 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:44.842167 2559 generic.go:356] "Generic (PLEG): container finished" podID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerID="38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545" exitCode=0 Sep 16 19:08:44.842281 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:44.842240 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" event={"ID":"9562e8a4-b4f6-4ec1-8145-9403ca36f85d","Type":"ContainerDied","Data":"38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545"} Sep 16 19:08:45.848205 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.848168 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" event={"ID":"164271d6-2621-4e8c-a224-3c1e44dc24c7","Type":"ContainerStarted","Data":"00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b"} Sep 16 19:08:45.848669 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.848242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" event={"ID":"164271d6-2621-4e8c-a224-3c1e44dc24c7","Type":"ContainerStarted","Data":"e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1"} Sep 16 19:08:45.848669 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.848542 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:45.850310 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.850286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" event={"ID":"9562e8a4-b4f6-4ec1-8145-9403ca36f85d","Type":"ContainerStarted","Data":"ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441"} Sep 16 19:08:45.850424 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.850336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" event={"ID":"9562e8a4-b4f6-4ec1-8145-9403ca36f85d","Type":"ContainerStarted","Data":"4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9"} Sep 16 19:08:45.850666 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.850650 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:45.850707 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.850696 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:45.851949 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.851924 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:08:45.876632 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.876589 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podStartSLOduration=2.84109634 podStartE2EDuration="6.876575616s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.035479276s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:45.875424871 +0000 UTC m=+1413.674546722" watchObservedRunningTime="2026-09-16 19:08:45.876575616 +0000 UTC m=+1413.675697455" Sep 16 19:08:45.911373 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:45.911323 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podStartSLOduration=3.110434236 podStartE2EDuration="6.911308744s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.800874508s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:08:45.910612449 +0000 UTC m=+1413.709734319" watchObservedRunningTime="2026-09-16 19:08:45.911308744 +0000 UTC m=+1413.710430583" Sep 16 19:08:46.854875 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:46.854836 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:08:46.855340 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:46.854938 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:46.856232 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:46.856207 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:08:47.858280 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:47.858243 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:08:51.859904 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:51.859874 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:08:51.860361 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:51.860335 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:08:52.862878 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:52.862845 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:08:52.863334 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:08:52.863308 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:09:00.960655 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:00.960613 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:01.860755 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:01.860715 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:09:02.864271 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:02.864231 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:09:11.860864 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:11.860775 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:09:12.864013 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:12.863951 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:09:21.860796 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:21.860755 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:09:22.863339 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:22.863299 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:09:27.045806 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:27.045770 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:09:31.860896 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:31.860858 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:09:32.864129 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:32.864090 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:09:41.861221 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:41.861184 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:09:42.863271 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:42.863227 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:09:51.860700 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:51.860656 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:09:52.864219 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:52.864180 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:09:54.717066 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:09:54.717024 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused" Sep 16 19:10:02.864068 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:02.864041 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:10:02.993912 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:02.993880 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf"] Sep 16 19:10:02.994179 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:02.994153 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" containerID="cri-o://4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9" gracePeriod=30 Sep 16 19:10:02.994250 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:02.994203 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kube-rbac-proxy" containerID="cri-o://ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441" gracePeriod=30 Sep 16 19:10:02.995036 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:02.995014 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:10:03.137753 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:03.137666 2559 generic.go:356] "Generic (PLEG): container finished" podID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerID="ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441" exitCode=2 Sep 16 19:10:03.137753 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:03.137733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" event={"ID":"9562e8a4-b4f6-4ec1-8145-9403ca36f85d","Type":"ContainerDied","Data":"ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441"} Sep 16 19:10:06.855255 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:06.855208 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused" Sep 16 19:10:07.635156 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.635128 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:10:07.685019 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.684920 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p6pmj\" (UniqueName: \"kubernetes.io/projected/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-api-access-p6pmj\") pod \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " Sep 16 19:10:07.685019 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.684980 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-cabundle-cert\") pod \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " Sep 16 19:10:07.685019 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.685009 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls\") pod \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " Sep 16 19:10:07.685271 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.685043 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kserve-provision-location\") pod \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " Sep 16 19:10:07.685271 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.685084 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-rbac-proxy-sar-config\") pod \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\" (UID: \"9562e8a4-b4f6-4ec1-8145-9403ca36f85d\") " Sep 16 19:10:07.685433 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.685336 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9562e8a4-b4f6-4ec1-8145-9403ca36f85d" (UID: "9562e8a4-b4f6-4ec1-8145-9403ca36f85d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:10:07.685495 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.685383 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9562e8a4-b4f6-4ec1-8145-9403ca36f85d" (UID: "9562e8a4-b4f6-4ec1-8145-9403ca36f85d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:10:07.685549 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.685508 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9562e8a4-b4f6-4ec1-8145-9403ca36f85d" (UID: "9562e8a4-b4f6-4ec1-8145-9403ca36f85d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:10:07.687132 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.687113 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9562e8a4-b4f6-4ec1-8145-9403ca36f85d" (UID: "9562e8a4-b4f6-4ec1-8145-9403ca36f85d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:10:07.687132 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.687119 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-api-access-p6pmj" (OuterVolumeSpecName: "kube-api-access-p6pmj") pod "9562e8a4-b4f6-4ec1-8145-9403ca36f85d" (UID: "9562e8a4-b4f6-4ec1-8145-9403ca36f85d"). InnerVolumeSpecName "kube-api-access-p6pmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:10:07.785804 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.785744 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:10:07.785804 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.785795 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:10:07.785804 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.785807 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6pmj\" (UniqueName: \"kubernetes.io/projected/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-kube-api-access-p6pmj\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:10:07.786080 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.785822 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:10:07.786080 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:07.785835 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9562e8a4-b4f6-4ec1-8145-9403ca36f85d-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:10:08.157113 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.157042 2559 generic.go:356] "Generic (PLEG): container finished" podID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerID="4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9" exitCode=0 Sep 16 19:10:08.157113 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.157097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" event={"ID":"9562e8a4-b4f6-4ec1-8145-9403ca36f85d","Type":"ContainerDied","Data":"4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9"} Sep 16 19:10:08.157519 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.157140 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" event={"ID":"9562e8a4-b4f6-4ec1-8145-9403ca36f85d","Type":"ContainerDied","Data":"2818f7ec712d94c927a115323401427882f3233136802c0b6e7311bbfc23565d"} Sep 16 19:10:08.157519 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.157165 2559 scope.go:117] "RemoveContainer" containerID="ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441" Sep 16 19:10:08.157519 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.157100 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf" Sep 16 19:10:08.165439 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.165421 2559 scope.go:117] "RemoveContainer" containerID="4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9" Sep 16 19:10:08.172672 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.172651 2559 scope.go:117] "RemoveContainer" containerID="38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545" Sep 16 19:10:08.179397 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.179375 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf"] Sep 16 19:10:08.180071 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.180056 2559 scope.go:117] "RemoveContainer" containerID="ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441" Sep 16 19:10:08.180342 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:10:08.180324 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441\": container with ID starting with ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441 not found: ID does not exist" containerID="ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441" Sep 16 19:10:08.180394 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.180351 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441"} err="failed to get container status \"ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441\": rpc error: code = NotFound desc = could not find container \"ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441\": container with ID starting with ee8886f1920b7af5ae45b325ac1be5a89547f708fc07a1d27edbed3e2f92e441 not found: ID does not exist" Sep 16 19:10:08.180394 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.180369 2559 scope.go:117] "RemoveContainer" containerID="4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9" Sep 16 19:10:08.180619 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:10:08.180601 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9\": container with ID starting with 4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9 not found: ID does not exist" containerID="4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9" Sep 16 19:10:08.180663 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.180625 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9"} err="failed to get container status \"4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9\": rpc error: code = NotFound desc = could not find container \"4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9\": container with ID starting with 4991ac4b4558ae930483f73d99b08cfd94474d39979249df05b5319c73b9cdd9 not found: ID does not exist" Sep 16 19:10:08.180663 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.180641 2559 scope.go:117] "RemoveContainer" containerID="38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545" Sep 16 19:10:08.180853 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:10:08.180833 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545\": container with ID starting with 38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545 not found: ID does not exist" containerID="38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545" Sep 16 19:10:08.180905 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.180862 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545"} err="failed to get container status \"38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545\": rpc error: code = NotFound desc = could not find container \"38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545\": container with ID starting with 38a82eeded5b84f00453d58081cb3d1e566ed23f833dfbf0d90ec1ca22942545 not found: ID does not exist" Sep 16 19:10:08.188503 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.188474 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-2lzlf"] Sep 16 19:10:08.721136 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:08.721101 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" path="/var/lib/kubelet/pods/9562e8a4-b4f6-4ec1-8145-9403ca36f85d/volumes" Sep 16 19:10:09.535433 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:09.535403 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:10:12.680020 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:12.679997 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 19:10:12.683037 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:10:12.683018 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 19:11:39.671512 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:39.671470 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk"] Sep 16 19:11:39.673978 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:39.672020 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" containerID="cri-o://e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1" gracePeriod=30 Sep 16 19:11:39.673978 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:39.672031 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kube-rbac-proxy" containerID="cri-o://00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b" gracePeriod=30 Sep 16 19:11:40.493298 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:40.493263 2559 generic.go:356] "Generic (PLEG): container finished" podID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerID="00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b" exitCode=2 Sep 16 19:11:40.493458 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:40.493315 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" event={"ID":"164271d6-2621-4e8c-a224-3c1e44dc24c7","Type":"ContainerDied","Data":"00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b"} Sep 16 19:11:40.872722 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:40.872646 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:42.859143 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:42.859104 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused" Sep 16 19:11:42.863462 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:42.863436 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused" Sep 16 19:11:44.209466 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.209443 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:11:44.261050 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.261019 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/164271d6-2621-4e8c-a224-3c1e44dc24c7-kserve-provision-location\") pod \"164271d6-2621-4e8c-a224-3c1e44dc24c7\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " Sep 16 19:11:44.261244 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.261076 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-cabundle-cert\") pod \"164271d6-2621-4e8c-a224-3c1e44dc24c7\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " Sep 16 19:11:44.261244 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.261112 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-rbac-proxy-sar-config\") pod \"164271d6-2621-4e8c-a224-3c1e44dc24c7\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " Sep 16 19:11:44.261244 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.261147 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54ppf\" (UniqueName: \"kubernetes.io/projected/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-api-access-54ppf\") pod \"164271d6-2621-4e8c-a224-3c1e44dc24c7\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " Sep 16 19:11:44.261244 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.261213 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls\") pod \"164271d6-2621-4e8c-a224-3c1e44dc24c7\" (UID: \"164271d6-2621-4e8c-a224-3c1e44dc24c7\") " Sep 16 19:11:44.261460 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.261400 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164271d6-2621-4e8c-a224-3c1e44dc24c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "164271d6-2621-4e8c-a224-3c1e44dc24c7" (UID: "164271d6-2621-4e8c-a224-3c1e44dc24c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:11:44.261460 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.261434 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "164271d6-2621-4e8c-a224-3c1e44dc24c7" (UID: "164271d6-2621-4e8c-a224-3c1e44dc24c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:11:44.261748 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.261721 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "164271d6-2621-4e8c-a224-3c1e44dc24c7" (UID: "164271d6-2621-4e8c-a224-3c1e44dc24c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:11:44.263224 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.263196 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "164271d6-2621-4e8c-a224-3c1e44dc24c7" (UID: "164271d6-2621-4e8c-a224-3c1e44dc24c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:11:44.263309 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.263226 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-api-access-54ppf" (OuterVolumeSpecName: "kube-api-access-54ppf") pod "164271d6-2621-4e8c-a224-3c1e44dc24c7" (UID: "164271d6-2621-4e8c-a224-3c1e44dc24c7"). InnerVolumeSpecName "kube-api-access-54ppf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:11:44.362285 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.362208 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:11:44.362285 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.362243 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:11:44.362285 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.362255 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54ppf\" (UniqueName: \"kubernetes.io/projected/164271d6-2621-4e8c-a224-3c1e44dc24c7-kube-api-access-54ppf\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:11:44.362285 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.362264 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/164271d6-2621-4e8c-a224-3c1e44dc24c7-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:11:44.362285 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.362271 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/164271d6-2621-4e8c-a224-3c1e44dc24c7-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:11:44.509752 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.509720 2559 generic.go:356] "Generic (PLEG): container finished" podID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerID="e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1" exitCode=0 Sep 16 19:11:44.509924 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.509774 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" Sep 16 19:11:44.509924 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.509810 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" event={"ID":"164271d6-2621-4e8c-a224-3c1e44dc24c7","Type":"ContainerDied","Data":"e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1"} Sep 16 19:11:44.509924 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.509883 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk" event={"ID":"164271d6-2621-4e8c-a224-3c1e44dc24c7","Type":"ContainerDied","Data":"7dc58ea9d349cb694f7ae909e04a540d371baeb3a149b2529fde39a8a77adb2d"} Sep 16 19:11:44.509924 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.509920 2559 scope.go:117] "RemoveContainer" containerID="00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b" Sep 16 19:11:44.518592 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.518573 2559 scope.go:117] "RemoveContainer" containerID="e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1" Sep 16 19:11:44.525409 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.525393 2559 scope.go:117] "RemoveContainer" containerID="aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40" Sep 16 19:11:44.532445 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.532422 2559 scope.go:117] "RemoveContainer" containerID="00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b" Sep 16 19:11:44.532740 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:11:44.532723 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b\": container with ID starting with 00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b not found: ID does not exist" containerID="00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b" Sep 16 19:11:44.532793 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.532748 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b"} err="failed to get container status \"00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b\": rpc error: code = NotFound desc = could not find container \"00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b\": container with ID starting with 00deeb2e7686de938fa72fea05d1873a03b05589e7dd2aa1e59078bdd06cf60b not found: ID does not exist" Sep 16 19:11:44.532793 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.532764 2559 scope.go:117] "RemoveContainer" containerID="e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1" Sep 16 19:11:44.533008 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:11:44.532993 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1\": container with ID starting with e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1 not found: ID does not exist" containerID="e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1" Sep 16 19:11:44.533054 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.533012 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1"} err="failed to get container status \"e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1\": rpc error: code = NotFound desc = could not find container \"e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1\": container with ID starting with e3ee87582835241ecb33efd62bd7d5b18cba9db875ed3e273c1bacb4e9f427d1 not found: ID does not exist" Sep 16 19:11:44.533054 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.533024 2559 scope.go:117] "RemoveContainer" containerID="aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40" Sep 16 19:11:44.533135 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.533074 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk"] Sep 16 19:11:44.533239 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:11:44.533225 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40\": container with ID starting with aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40 not found: ID does not exist" containerID="aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40" Sep 16 19:11:44.533284 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.533242 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40"} err="failed to get container status \"aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40\": rpc error: code = NotFound desc = could not find container \"aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40\": container with ID starting with aa14f3ebacce56cdd959fd1065479379fc345b50ebdcf653318df32bb7b22e40 not found: ID does not exist" Sep 16 19:11:44.536521 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.536501 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-j4lpk"] Sep 16 19:11:44.720844 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:44.720819 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" path="/var/lib/kubelet/pods/164271d6-2621-4e8c-a224-3c1e44dc24c7/volumes" Sep 16 19:11:54.276937 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:54.276878 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:11:54.426898 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:11:54.426871 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:14.376799 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.376763 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt"] Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377013 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="storage-initializer" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377026 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="storage-initializer" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377033 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377039 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377047 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377053 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377062 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kube-rbac-proxy" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377067 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kube-rbac-proxy" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377076 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="storage-initializer" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377081 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="storage-initializer" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377091 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kube-rbac-proxy" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377096 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kube-rbac-proxy" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377134 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kserve-container" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377139 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kserve-container" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377147 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9562e8a4-b4f6-4ec1-8145-9403ca36f85d" containerName="kube-rbac-proxy" Sep 16 19:12:14.377177 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.377153 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="164271d6-2621-4e8c-a224-3c1e44dc24c7" containerName="kube-rbac-proxy" Sep 16 19:12:14.380083 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.380067 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.385659 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.385638 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 19:12:14.386399 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.386376 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\"" Sep 16 19:12:14.386510 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.386408 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 19:12:14.386510 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.386413 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-4znnq\"" Sep 16 19:12:14.386625 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.386560 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 19:12:14.387059 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.387038 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-cipn-predictor-serving-cert\"" Sep 16 19:12:14.388228 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.388212 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 19:12:14.391746 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.391728 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:14.403205 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.403183 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt"] Sep 16 19:12:14.474197 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.474167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/473166d4-493a-4776-9fc5-9e2deb6a3e22-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.474370 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.474203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.474370 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.474236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/473166d4-493a-4776-9fc5-9e2deb6a3e22-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.474370 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.474337 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.474515 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.474386 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z76pz\" (UniqueName: \"kubernetes.io/projected/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-api-access-z76pz\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.575383 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.575350 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.575556 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.575392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z76pz\" (UniqueName: \"kubernetes.io/projected/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-api-access-z76pz\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.575556 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.575414 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/473166d4-493a-4776-9fc5-9e2deb6a3e22-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.575556 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.575442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.575556 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.575474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/473166d4-493a-4776-9fc5-9e2deb6a3e22-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.575837 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.575817 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/473166d4-493a-4776-9fc5-9e2deb6a3e22-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.576069 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.576047 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.576161 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.576145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.577813 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.577796 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/473166d4-493a-4776-9fc5-9e2deb6a3e22-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.584763 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.584738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z76pz\" (UniqueName: \"kubernetes.io/projected/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-api-access-z76pz\") pod \"isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.587913 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.587893 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:14.691244 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.691222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:14.983845 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:14.983819 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt"] Sep 16 19:12:14.985592 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:12:14.985563 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod473166d4_493a_4776_9fc5_9e2deb6a3e22.slice/crio-70bd80d5d422c7a04984c2902602da9bacc764b513cab0172992f5d6b06ad7b4 WatchSource:0}: Error finding container 70bd80d5d422c7a04984c2902602da9bacc764b513cab0172992f5d6b06ad7b4: Status 404 returned error can't find the container with id 70bd80d5d422c7a04984c2902602da9bacc764b513cab0172992f5d6b06ad7b4 Sep 16 19:12:15.626038 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:15.626000 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerStarted","Data":"08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f"} Sep 16 19:12:15.626465 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:15.626083 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerStarted","Data":"70bd80d5d422c7a04984c2902602da9bacc764b513cab0172992f5d6b06ad7b4"} Sep 16 19:12:19.642703 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:19.642670 2559 generic.go:356] "Generic (PLEG): container finished" podID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerID="08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f" exitCode=0 Sep 16 19:12:19.643098 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:19.642749 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerDied","Data":"08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f"} Sep 16 19:12:20.648319 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:20.648284 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerStarted","Data":"5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64"} Sep 16 19:12:20.648773 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:20.648358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerStarted","Data":"f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7"} Sep 16 19:12:20.648773 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:20.648401 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerStarted","Data":"060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08"} Sep 16 19:12:20.648773 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:20.648644 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:20.697731 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:20.697683 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podStartSLOduration=3.008532537 podStartE2EDuration="6.697664941s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.689132404s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:20.696511957 +0000 UTC m=+1628.495633813" watchObservedRunningTime="2026-09-16 19:12:20.697664941 +0000 UTC m=+1628.496786781" Sep 16 19:12:21.652276 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:21.652241 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:21.652630 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:21.652299 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:21.653678 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:21.653649 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:12:21.654366 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:21.654343 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:12:22.655613 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:22.655568 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:12:22.656028 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:22.655850 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:12:26.480788 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:26.480755 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:27.660165 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:27.660133 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:12:27.660688 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:27.660649 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:12:27.660940 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:27.660916 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:12:32.899562 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:32.899531 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:37.030835 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:37.030788 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:37.661432 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:37.661396 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:12:37.661938 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:37.661915 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:12:47.040521 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:47.040485 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:12:47.661282 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:47.661236 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:12:47.661674 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:47.661649 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:12:57.660893 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:57.660847 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:12:57.661361 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:12:57.661330 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:13:07.661377 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:07.661331 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:13:07.661840 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:07.661815 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:13:12.759276 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:12.755761 2559 scope.go:117] "RemoveContainer" containerID="4be8bac7643063c224ec1580944dffce959b121521f493718e82b2ea17919da7" Sep 16 19:13:12.767257 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:12.767235 2559 scope.go:117] "RemoveContainer" containerID="d79e6b2f197d83aafa54893b912b035ffaa992431041e2441b5cc310598a772d" Sep 16 19:13:12.774394 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:12.774368 2559 scope.go:117] "RemoveContainer" containerID="559165becf92782ab3189108f38890841d080e464897f70fa5e003e9cee36530" Sep 16 19:13:17.660902 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:17.660857 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:13:17.661447 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:17.661292 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:13:27.661139 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:27.661102 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:13:27.661644 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:27.661622 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:13:37.661869 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:37.661789 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:13:37.662368 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:37.661950 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:13:49.550252 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:49.550217 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt"] Sep 16 19:13:49.550815 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:49.550489 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" containerID="cri-o://060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08" gracePeriod=30 Sep 16 19:13:49.550815 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:49.550518 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" containerID="cri-o://f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7" gracePeriod=30 Sep 16 19:13:49.550815 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:49.550519 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" containerID="cri-o://5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64" gracePeriod=30 Sep 16 19:13:49.603750 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:49.603716 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:49.963396 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:49.963362 2559 generic.go:356] "Generic (PLEG): container finished" podID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerID="f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7" exitCode=2 Sep 16 19:13:49.963563 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:49.963436 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerDied","Data":"f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7"} Sep 16 19:13:51.163110 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:51.163074 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:13:52.655622 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:52.655581 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 16 19:13:53.978576 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:53.978547 2559 generic.go:356] "Generic (PLEG): container finished" podID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerID="060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08" exitCode=0 Sep 16 19:13:53.978882 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:53.978616 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerDied","Data":"060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08"} Sep 16 19:13:57.656377 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:57.656331 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 16 19:13:57.661601 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:57.661559 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:13:57.661900 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:13:57.661877 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:14:02.656373 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:02.656331 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 16 19:14:02.656758 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:02.656459 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:14:06.224465 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:06.224438 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:06.250425 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:06.250393 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:06.263374 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:06.263354 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:06.275415 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:06.275369 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:07.081035 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.081004 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:07.092738 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.092701 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:07.104674 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.104648 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:07.116280 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.116265 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:07.656250 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.656211 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 16 19:14:07.660604 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.660572 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:14:07.660912 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.660893 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:14:07.948176 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.948148 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:07.964908 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.964884 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:07.979680 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.979661 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:07.990259 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:07.990236 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:08.795100 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:08.795071 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:08.807510 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:08.807489 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:08.817920 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:08.817899 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:08.828914 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:08.828896 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:09.669493 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:09.669459 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:09.682651 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:09.682620 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:09.701563 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:09.701536 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:09.710009 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:09.709993 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:10.512817 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:10.512788 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:10.523097 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:10.523077 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:10.534375 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:10.534356 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:10.559841 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:10.559822 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:11.376872 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:11.376842 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:11.391068 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:11.391045 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:11.401033 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:11.401007 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:11.411624 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:11.411602 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:12.196952 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:12.196923 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:12.208650 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:12.208608 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:12.217876 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:12.217843 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:12.226905 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:12.226879 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:12.656725 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:12.656631 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 16 19:14:13.034724 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:13.034694 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:13.045475 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:13.045445 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:13.054648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:13.054621 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:13.063754 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:13.063729 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:13.872569 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:13.872541 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:13.883818 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:13.883795 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:13.903832 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:13.903811 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:13.918465 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:13.918445 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:14.714858 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:14.714828 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:14.726289 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:14.726263 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:14.737451 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:14.737425 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:14.748325 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:14.748304 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:15.554542 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:15.554512 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:15.565337 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:15.565309 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:15.574597 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:15.574574 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:15.584803 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:15.584777 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:16.380977 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:16.380939 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:16.390847 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:16.390827 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:16.401347 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:16.401329 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:16.411432 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:16.411415 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:17.214110 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.214079 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:17.238055 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.238031 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:17.254091 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.254069 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:17.275564 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.275543 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:17.656091 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.655998 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused" Sep 16 19:14:17.661392 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.661366 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused" Sep 16 19:14:17.661487 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.661475 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:14:17.661779 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.661755 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 16 19:14:17.661876 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:17.661862 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:14:18.280443 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:18.280419 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kserve-container/0.log" Sep 16 19:14:18.290902 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:18.290883 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/kube-rbac-proxy/0.log" Sep 16 19:14:18.300354 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:18.300336 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/agent/0.log" Sep 16 19:14:18.309243 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:18.309225 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt_473166d4-493a-4776-9fc5-9e2deb6a3e22/storage-initializer/0.log" Sep 16 19:14:19.719602 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.719581 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:14:19.778542 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.778515 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-cabundle-cert\") pod \"473166d4-493a-4776-9fc5-9e2deb6a3e22\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " Sep 16 19:14:19.778696 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.778551 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z76pz\" (UniqueName: \"kubernetes.io/projected/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-api-access-z76pz\") pod \"473166d4-493a-4776-9fc5-9e2deb6a3e22\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " Sep 16 19:14:19.778696 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.778570 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-rbac-proxy-sar-config\") pod \"473166d4-493a-4776-9fc5-9e2deb6a3e22\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " Sep 16 19:14:19.778696 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.778617 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/473166d4-493a-4776-9fc5-9e2deb6a3e22-kserve-provision-location\") pod \"473166d4-493a-4776-9fc5-9e2deb6a3e22\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " Sep 16 19:14:19.778696 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.778650 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/473166d4-493a-4776-9fc5-9e2deb6a3e22-proxy-tls\") pod \"473166d4-493a-4776-9fc5-9e2deb6a3e22\" (UID: \"473166d4-493a-4776-9fc5-9e2deb6a3e22\") " Sep 16 19:14:19.778938 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.778920 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "473166d4-493a-4776-9fc5-9e2deb6a3e22" (UID: "473166d4-493a-4776-9fc5-9e2deb6a3e22"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:14:19.779016 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.778942 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "473166d4-493a-4776-9fc5-9e2deb6a3e22" (UID: "473166d4-493a-4776-9fc5-9e2deb6a3e22"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 19:14:19.779016 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.778984 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/473166d4-493a-4776-9fc5-9e2deb6a3e22-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "473166d4-493a-4776-9fc5-9e2deb6a3e22" (UID: "473166d4-493a-4776-9fc5-9e2deb6a3e22"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:14:19.780668 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.780649 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/473166d4-493a-4776-9fc5-9e2deb6a3e22-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "473166d4-493a-4776-9fc5-9e2deb6a3e22" (UID: "473166d4-493a-4776-9fc5-9e2deb6a3e22"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 19:14:19.780730 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.780651 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-api-access-z76pz" (OuterVolumeSpecName: "kube-api-access-z76pz") pod "473166d4-493a-4776-9fc5-9e2deb6a3e22" (UID: "473166d4-493a-4776-9fc5-9e2deb6a3e22"). InnerVolumeSpecName "kube-api-access-z76pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:14:19.879188 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.879089 2559 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-cabundle-cert\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:14:19.879188 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.879123 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z76pz\" (UniqueName: \"kubernetes.io/projected/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-api-access-z76pz\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:14:19.879188 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.879137 2559 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/473166d4-493a-4776-9fc5-9e2deb6a3e22-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:14:19.879188 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.879149 2559 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/473166d4-493a-4776-9fc5-9e2deb6a3e22-kserve-provision-location\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:14:19.879188 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:19.879163 2559 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/473166d4-493a-4776-9fc5-9e2deb6a3e22-proxy-tls\") on node \"ip-10-0-140-230.ec2.internal\" DevicePath \"\"" Sep 16 19:14:20.072596 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.072561 2559 generic.go:356] "Generic (PLEG): container finished" podID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerID="5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64" exitCode=0 Sep 16 19:14:20.072795 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.072631 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" Sep 16 19:14:20.072795 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.072635 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerDied","Data":"5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64"} Sep 16 19:14:20.072795 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.072762 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt" event={"ID":"473166d4-493a-4776-9fc5-9e2deb6a3e22","Type":"ContainerDied","Data":"70bd80d5d422c7a04984c2902602da9bacc764b513cab0172992f5d6b06ad7b4"} Sep 16 19:14:20.072795 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.072793 2559 scope.go:117] "RemoveContainer" containerID="5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64" Sep 16 19:14:20.080874 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.080855 2559 scope.go:117] "RemoveContainer" containerID="f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7" Sep 16 19:14:20.087800 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.087783 2559 scope.go:117] "RemoveContainer" containerID="060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08" Sep 16 19:14:20.094422 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.094394 2559 scope.go:117] "RemoveContainer" containerID="08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f" Sep 16 19:14:20.096692 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.096672 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt"] Sep 16 19:14:20.101465 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.101445 2559 scope.go:117] "RemoveContainer" containerID="5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64" Sep 16 19:14:20.101894 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:14:20.101807 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64\": container with ID starting with 5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64 not found: ID does not exist" containerID="5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64" Sep 16 19:14:20.101894 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.101860 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64"} err="failed to get container status \"5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64\": rpc error: code = NotFound desc = could not find container \"5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64\": container with ID starting with 5360f1714cb61a4e8e9a9962a41cfc7335133e3964633bb8e548bba28a301f64 not found: ID does not exist" Sep 16 19:14:20.101894 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.101886 2559 scope.go:117] "RemoveContainer" containerID="f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7" Sep 16 19:14:20.102208 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:14:20.102191 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7\": container with ID starting with f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7 not found: ID does not exist" containerID="f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7" Sep 16 19:14:20.102253 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.102214 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7"} err="failed to get container status \"f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7\": rpc error: code = NotFound desc = could not find container \"f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7\": container with ID starting with f44fb04ac27fc3313f742192c5baaa4a806a996b83f583af16a81bca608c81e7 not found: ID does not exist" Sep 16 19:14:20.102253 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.102230 2559 scope.go:117] "RemoveContainer" containerID="060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08" Sep 16 19:14:20.102485 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:14:20.102467 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08\": container with ID starting with 060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08 not found: ID does not exist" containerID="060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08" Sep 16 19:14:20.102528 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.102492 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08"} err="failed to get container status \"060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08\": rpc error: code = NotFound desc = could not find container \"060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08\": container with ID starting with 060b39cdc5456ca40cd9f9a7c072174a6b90eb2a763fa45d44d5dbf1fbe79a08 not found: ID does not exist" Sep 16 19:14:20.102528 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.102507 2559 scope.go:117] "RemoveContainer" containerID="08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f" Sep 16 19:14:20.102771 ip-10-0-140-230 kubenswrapper[2559]: E0916 19:14:20.102752 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f\": container with ID starting with 08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f not found: ID does not exist" containerID="08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f" Sep 16 19:14:20.102825 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.102780 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f"} err="failed to get container status \"08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f\": rpc error: code = NotFound desc = could not find container \"08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f\": container with ID starting with 08c6eb9e3f667a9aeb06c00d9a9c8bf4af4f7075754222eae690cf596301282f not found: ID does not exist" Sep 16 19:14:20.103095 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.103079 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-cipn-predictor-67b78f46c9-dhsmt"] Sep 16 19:14:20.721722 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:20.721687 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" path="/var/lib/kubelet/pods/473166d4-493a-4776-9fc5-9e2deb6a3e22/volumes" Sep 16 19:14:22.486806 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:22.486773 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:22.628351 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:22.628320 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:22.749564 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:22.749476 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:25.378055 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:25.378023 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sz2r4_79ae6a44-0eb1-43cc-a9ff-63bc5342a979/global-pull-secret-syncer/0.log" Sep 16 19:14:25.415273 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:25.415248 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" Sep 16 19:14:25.514651 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:25.514626 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-230.ec2.internal_d512deac78b026258432227d6ed92f58/haproxy/0.log" Sep 16 19:14:28.180261 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:28.180229 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:28.439713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:28.439639 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f9pjr_9ccc0ada-ccbb-465c-9ae5-fd0aab1219dc/konnectivity-agent/0.log" limit=1024 Sep 16 19:14:29.510213 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:29.510186 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkhsn_a551dd06-0827-481f-9229-41f146c34ac3/node-exporter/0.log" Sep 16 19:14:29.552915 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:29.552888 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkhsn_a551dd06-0827-481f-9229-41f146c34ac3/kube-rbac-proxy/0.log" Sep 16 19:14:29.579711 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:29.579688 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nkhsn_a551dd06-0827-481f-9229-41f146c34ac3/init-textfile/0.log" Sep 16 19:14:33.865359 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865324 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg"] Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865589 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="storage-initializer" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865601 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="storage-initializer" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865619 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865625 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865635 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865641 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865647 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865652 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865692 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kube-rbac-proxy" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865699 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="agent" Sep 16 19:14:33.865713 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.865705 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="473166d4-493a-4776-9fc5-9e2deb6a3e22" containerName="kserve-container" Sep 16 19:14:33.869855 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.869837 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:33.872745 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.872729 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68wb5\"/\"kube-root-ca.crt\"" Sep 16 19:14:33.874120 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.874099 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-68wb5\"/\"default-dockercfg-fd5h5\"" Sep 16 19:14:33.874235 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.874177 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68wb5\"/\"openshift-service-ca.crt\"" Sep 16 19:14:33.878928 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.878905 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg"] Sep 16 19:14:33.964124 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.964089 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-lib-modules\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:33.964286 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.964135 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-podres\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:33.964286 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.964200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8p58\" (UniqueName: \"kubernetes.io/projected/d846a9ca-b760-49b1-a6f0-64062d72409b-kube-api-access-v8p58\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:33.964286 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.964245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-sys\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:33.964286 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:33.964263 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-proc\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.064960 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.064926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-lib-modules\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.064960 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.064977 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-podres\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.065223 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.065100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8p58\" (UniqueName: \"kubernetes.io/projected/d846a9ca-b760-49b1-a6f0-64062d72409b-kube-api-access-v8p58\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.065223 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.065130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-podres\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.065223 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.065099 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-lib-modules\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.065223 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.065154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-sys\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.065223 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.065198 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-proc\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.065223 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.065203 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-sys\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.065462 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.065276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d846a9ca-b760-49b1-a6f0-64062d72409b-proc\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.073818 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.073797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8p58\" (UniqueName: \"kubernetes.io/projected/d846a9ca-b760-49b1-a6f0-64062d72409b-kube-api-access-v8p58\") pod \"perf-node-gather-daemonset-6ncpg\" (UID: \"d846a9ca-b760-49b1-a6f0-64062d72409b\") " pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.179855 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.179823 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:34.367897 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.367863 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmp8z_aed88e62-448f-471e-8680-79d5d962b285/dns/0.log" Sep 16 19:14:34.398586 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.398558 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmp8z_aed88e62-448f-471e-8680-79d5d962b285/kube-rbac-proxy/0.log" Sep 16 19:14:34.453696 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.453613 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg"] Sep 16 19:14:34.457922 ip-10-0-140-230 kubenswrapper[2559]: W0916 19:14:34.457887 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd846a9ca_b760_49b1_a6f0_64062d72409b.slice/crio-e0af8c8b616c9340e77976ba773ddb0a847c251be55cb78de793f8efb932b734 WatchSource:0}: Error finding container e0af8c8b616c9340e77976ba773ddb0a847c251be55cb78de793f8efb932b734: Status 404 returned error can't find the container with id e0af8c8b616c9340e77976ba773ddb0a847c251be55cb78de793f8efb932b734 Sep 16 19:14:34.618151 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:34.618122 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-57pgm_956fd3b6-79ea-4195-8ce8-35658bd3d7de/dns-node-resolver/0.log" Sep 16 19:14:35.126256 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:35.126220 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" event={"ID":"d846a9ca-b760-49b1-a6f0-64062d72409b","Type":"ContainerStarted","Data":"491982e1385908b36f0642908174c59181997ce7989dadd50e49c34a7d0339d2"} Sep 16 19:14:35.126724 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:35.126277 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" event={"ID":"d846a9ca-b760-49b1-a6f0-64062d72409b","Type":"ContainerStarted","Data":"e0af8c8b616c9340e77976ba773ddb0a847c251be55cb78de793f8efb932b734"} Sep 16 19:14:35.126724 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:35.126345 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:35.148662 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:35.148620 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" podStartSLOduration=2.148606483 podStartE2EDuration="2.148606483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:35.148008008 +0000 UTC m=+1762.947129858" watchObservedRunningTime="2026-09-16 19:14:35.148606483 +0000 UTC m=+1762.947728322" Sep 16 19:14:35.210505 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:35.210478 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m9mfd_227b69aa-2d1b-4bf7-8603-f19fb4f2a606/node-ca/0.log" Sep 16 19:14:36.345991 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:36.345937 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cq5vr_9adce71e-476d-4ff2-9cd0-ce06e831d479/serve-healthcheck-canary/0.log" Sep 16 19:14:36.776204 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:36.776178 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lbj8_0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0/kube-rbac-proxy/0.log" Sep 16 19:14:36.798766 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:36.798744 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lbj8_0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0/exporter/0.log" Sep 16 19:14:36.820191 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:36.820171 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9lbj8_0c4c5923-5c42-4f52-99ec-06d2d0ccc4b0/extractor/0.log" Sep 16 19:14:39.237581 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:39.237554 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-75c4c6c467-qx55p_ee1560d2-5746-4918-bcdb-8e95be0526ab/manager/0.log" Sep 16 19:14:39.288635 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:39.288603 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-n95hw_ef6bff9d-15ba-4bd7-9e55-2f3917d77c8d/server/0.log" Sep 16 19:14:41.138056 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:41.138026 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-68wb5/perf-node-gather-daemonset-6ncpg" Sep 16 19:14:45.540390 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:45.540358 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9gp8g_46efde69-5c9c-4fee-956d-f2c2c5855225/kube-multus/0.log" Sep 16 19:14:46.355313 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.355270 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvmpj_4fa27ea5-e7d8-4be1-9c00-410f85826d80/kube-multus-additional-cni-plugins/0.log" Sep 16 19:14:46.391922 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.391876 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvmpj_4fa27ea5-e7d8-4be1-9c00-410f85826d80/egress-router-binary-copy/0.log" Sep 16 19:14:46.418553 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.418525 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvmpj_4fa27ea5-e7d8-4be1-9c00-410f85826d80/cni-plugins/0.log" Sep 16 19:14:46.443591 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.443562 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvmpj_4fa27ea5-e7d8-4be1-9c00-410f85826d80/bond-cni-plugin/0.log" Sep 16 19:14:46.469033 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.469008 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvmpj_4fa27ea5-e7d8-4be1-9c00-410f85826d80/routeoverride-cni/0.log" Sep 16 19:14:46.506408 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.506383 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvmpj_4fa27ea5-e7d8-4be1-9c00-410f85826d80/whereabouts-cni-bincopy/0.log" Sep 16 19:14:46.541154 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.541121 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xvmpj_4fa27ea5-e7d8-4be1-9c00-410f85826d80/whereabouts-cni/0.log" Sep 16 19:14:46.752690 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.752668 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x7nkj_3198a50c-14d2-4981-8d5d-7718876d54b1/network-metrics-daemon/0.log" Sep 16 19:14:46.785951 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:46.785919 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-x7nkj_3198a50c-14d2-4981-8d5d-7718876d54b1/kube-rbac-proxy/0.log" Sep 16 19:14:47.978009 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:47.977979 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-controller/0.log" Sep 16 19:14:47.996884 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:47.996861 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/1.log" Sep 16 19:14:47.997022 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:47.996894 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovn-acl-logging/0.log" Sep 16 19:14:48.015449 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:48.015425 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/kube-rbac-proxy-node/0.log" Sep 16 19:14:48.035459 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:48.035432 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 19:14:48.054550 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:48.054526 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/northd/0.log" Sep 16 19:14:48.074896 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:48.074875 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/nbdb/0.log" Sep 16 19:14:48.098648 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:48.098629 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/sbdb/0.log" Sep 16 19:14:48.135579 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:48.135555 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-v2qjb_2aef5ab2-8f8b-4a39-a986-7e7fb777a934/ovnkube-controller/0.log" Sep 16 19:14:49.316197 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:49.316172 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-6lmhf_127d816f-fcb2-4c92-aec9-b9c9228c7252/check-endpoints/0.log" Sep 16 19:14:49.367709 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:49.367685 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-jr5jk_db40c4ce-12ac-41c2-9b56-6a53b5616259/network-check-target-container/0.log" Sep 16 19:14:50.291105 ip-10-0-140-230 kubenswrapper[2559]: I0916 19:14:50.291077 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-dzfmw_1f6efc2c-af56-4997-92ec-93fbb8a3fb4e/iptables-alerter/0.log"