Sep 19 08:20:29.119316 ip-10-0-139-127 systemd[1]: Starting Kubernetes Kubelet... Sep 19 08:20:29.505689 ip-10-0-139-127 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 08:20:29.505689 ip-10-0-139-127 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 19 08:20:29.505689 ip-10-0-139-127 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 08:20:29.505689 ip-10-0-139-127 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 19 08:20:29.505689 ip-10-0-139-127 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 19 08:20:29.507324 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.507236 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 19 08:20:29.510073 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510057 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 08:20:29.510073 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510071 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 08:20:29.510073 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510075 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510078 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510081 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510084 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510087 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510090 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510093 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510095 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510098 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510101 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510103 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510106 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510109 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510114 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510117 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510121 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510124 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510127 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510129 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 08:20:29.510165 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510132 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510135 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510137 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510140 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510143 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510145 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510148 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510151 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510156 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510160 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510163 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510166 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510170 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510175 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510179 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510182 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510185 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510187 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510190 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 08:20:29.510616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510193 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510195 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510198 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510200 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510203 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510206 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510208 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510211 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510214 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510217 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510219 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510222 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510224 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510227 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510229 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510233 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510236 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510239 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510241 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 08:20:29.511090 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510244 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510246 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510249 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510252 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510255 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510258 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510260 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510263 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510266 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510268 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510271 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510274 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510276 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510279 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510281 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510284 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510287 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510290 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510292 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510295 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 08:20:29.511553 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510297 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510300 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510303 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510306 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510308 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510311 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510314 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510317 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510320 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510324 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510327 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510330 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510333 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510337 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510339 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510342 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510345 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510348 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510350 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.510353 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 08:20:29.512044 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511038 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511045 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511048 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511050 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511053 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511056 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511058 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511061 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511063 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511066 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511068 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511071 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511074 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511076 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511079 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511082 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511085 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511088 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511090 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 08:20:29.512520 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511093 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511097 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511100 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511104 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511108 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511111 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511113 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511116 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511118 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511121 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511124 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511126 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511129 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511132 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511134 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511137 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511140 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511143 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511145 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511148 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 08:20:29.513034 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511151 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511153 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511156 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511158 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511161 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511163 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511166 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511168 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511171 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511174 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511177 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511179 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511182 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511186 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511189 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511191 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511194 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511197 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511199 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511202 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 08:20:29.513531 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511204 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511207 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511210 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511213 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511215 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511218 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511220 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511223 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511225 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511228 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511230 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511233 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511235 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511238 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511242 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511247 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511250 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511252 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511255 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511258 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 08:20:29.514051 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511261 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511264 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511267 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511270 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511273 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511276 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511279 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511282 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511285 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511288 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511290 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511293 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511295 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511298 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511301 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511303 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511305 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511309 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511312 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511315 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 08:20:29.514550 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511676 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511681 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511684 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511687 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511690 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511693 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511697 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511700 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511703 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511705 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511708 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511711 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511713 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511716 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511719 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511722 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511725 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511728 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511731 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 08:20:29.515059 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511734 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511736 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511739 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511741 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511744 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511746 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511749 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511752 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511754 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511757 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511759 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511762 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511765 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511768 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511770 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511773 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511776 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511778 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511781 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511783 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 08:20:29.515521 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511788 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511791 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511794 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511797 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511800 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511802 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511805 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511807 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511811 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511813 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511816 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511818 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511821 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511824 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511827 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511829 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511833 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511835 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511838 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511840 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 08:20:29.515995 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511842 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511845 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511847 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511850 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511853 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511855 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511858 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511861 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511863 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511866 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511868 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511871 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511874 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511876 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511879 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511881 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511884 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511886 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511889 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511891 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 08:20:29.516569 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511894 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511897 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511900 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511903 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511905 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511908 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511910 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511913 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511915 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511918 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511921 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511924 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511926 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511928 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511932 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511936 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511939 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511942 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511945 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 08:20:29.517071 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.511948 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512034 2585 flags.go:64] FLAG: --address="0.0.0.0" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512066 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512072 2585 flags.go:64] FLAG: --anonymous-auth="true" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512077 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512083 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512087 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512091 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512096 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512099 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512102 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512105 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512109 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512112 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512115 2585 flags.go:64] FLAG: --cgroup-root="" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512118 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512121 2585 flags.go:64] FLAG: --client-ca-file="" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512124 2585 flags.go:64] FLAG: --cloud-provider="external" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512128 2585 flags.go:64] FLAG: --cluster-dns="[]" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512132 2585 flags.go:64] FLAG: --cluster-domain="" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512135 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512138 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512141 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512144 2585 flags.go:64] FLAG: --container-log-max-files="5" Sep 19 08:20:29.517529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512148 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512151 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512154 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512158 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512161 2585 flags.go:64] FLAG: --contention-profiling="false" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512163 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512166 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512170 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512172 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512177 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512180 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512182 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512185 2585 flags.go:64] FLAG: --enable-load-reader="false" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512190 2585 flags.go:64] FLAG: --enable-server="true" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512193 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512238 2585 flags.go:64] FLAG: --event-burst="100" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512241 2585 flags.go:64] FLAG: --event-qps="50" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512245 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512248 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512251 2585 flags.go:64] FLAG: --eviction-hard="" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512255 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512258 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512261 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512264 2585 flags.go:64] FLAG: --eviction-soft="" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512267 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 19 08:20:29.518171 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512270 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512273 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512276 2585 flags.go:64] FLAG: --experimental-mounter-path="" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512279 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512282 2585 flags.go:64] FLAG: --fail-swap-on="true" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512285 2585 flags.go:64] FLAG: --feature-gates="" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512288 2585 flags.go:64] FLAG: --file-check-frequency="20s" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512291 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512294 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512298 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512300 2585 flags.go:64] FLAG: --healthz-port="10248" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512304 2585 flags.go:64] FLAG: --help="false" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512307 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512310 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512313 2585 flags.go:64] FLAG: --http-check-frequency="20s" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512316 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512319 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512323 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512326 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512328 2585 flags.go:64] FLAG: --image-service-endpoint="" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512333 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512336 2585 flags.go:64] FLAG: --kube-api-burst="100" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512338 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512342 2585 flags.go:64] FLAG: --kube-api-qps="50" Sep 19 08:20:29.518784 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512345 2585 flags.go:64] FLAG: --kube-reserved="" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512347 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512350 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512353 2585 flags.go:64] FLAG: --kubelet-cgroups="" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512356 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512359 2585 flags.go:64] FLAG: --lock-file="" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512362 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512365 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512369 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512374 2585 flags.go:64] FLAG: --log-json-split-stream="false" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512377 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512381 2585 flags.go:64] FLAG: --log-text-split-stream="false" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512384 2585 flags.go:64] FLAG: --logging-format="text" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512387 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512390 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512393 2585 flags.go:64] FLAG: --manifest-url="" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512396 2585 flags.go:64] FLAG: --manifest-url-header="" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512400 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512403 2585 flags.go:64] FLAG: --max-open-files="1000000" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512407 2585 flags.go:64] FLAG: --max-pods="110" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512410 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512413 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512416 2585 flags.go:64] FLAG: --memory-manager-policy="None" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512419 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512422 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 19 08:20:29.519436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512425 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512428 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512435 2585 flags.go:64] FLAG: --node-status-max-images="50" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512439 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512442 2585 flags.go:64] FLAG: --oom-score-adj="-999" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512445 2585 flags.go:64] FLAG: --pod-cidr="" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512448 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512453 2585 flags.go:64] FLAG: --pod-manifest-path="" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512456 2585 flags.go:64] FLAG: --pod-max-pids="-1" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512459 2585 flags.go:64] FLAG: --pods-per-core="0" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512462 2585 flags.go:64] FLAG: --port="10250" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512465 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512468 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03d6d9fba80f68434" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512471 2585 flags.go:64] FLAG: --qos-reserved="" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512475 2585 flags.go:64] FLAG: --read-only-port="10255" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512478 2585 flags.go:64] FLAG: --register-node="true" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512481 2585 flags.go:64] FLAG: --register-with-taints="" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512485 2585 flags.go:64] FLAG: --registry-burst="10" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512491 2585 flags.go:64] FLAG: --registry-qps="5" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512494 2585 flags.go:64] FLAG: --reserved-cpus="" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512497 2585 flags.go:64] FLAG: --reserved-memory="" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512500 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512504 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512507 2585 flags.go:64] FLAG: --rotate-certificates="false" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512510 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 19 08:20:29.520140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512513 2585 flags.go:64] FLAG: --runonce="false" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512516 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512519 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512522 2585 flags.go:64] FLAG: --seccomp-default="false" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512525 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512528 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512531 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512534 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512537 2585 flags.go:64] FLAG: --storage-driver-password="root" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512540 2585 flags.go:64] FLAG: --storage-driver-secure="false" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512543 2585 flags.go:64] FLAG: --storage-driver-table="stats" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512548 2585 flags.go:64] FLAG: --storage-driver-user="root" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512551 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512554 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512557 2585 flags.go:64] FLAG: --system-cgroups="" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512560 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512565 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512568 2585 flags.go:64] FLAG: --tls-cert-file="" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512571 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512578 2585 flags.go:64] FLAG: --tls-min-version="" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512581 2585 flags.go:64] FLAG: --tls-private-key-file="" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512584 2585 flags.go:64] FLAG: --topology-manager-policy="none" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512587 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512590 2585 flags.go:64] FLAG: --topology-manager-scope="container" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512593 2585 flags.go:64] FLAG: --v="2" Sep 19 08:20:29.521067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512598 2585 flags.go:64] FLAG: --version="false" Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512602 2585 flags.go:64] FLAG: --vmodule="" Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512606 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.512609 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512742 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512745 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512749 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512751 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512754 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512757 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512760 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512763 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512766 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512769 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512772 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512774 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512777 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512779 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512783 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512786 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 08:20:29.521774 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512788 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512791 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512794 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512796 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512799 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512801 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512804 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512808 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512812 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512815 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512818 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512821 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512826 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512829 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512831 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512834 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512837 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512839 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512842 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 08:20:29.522330 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512844 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512847 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512851 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512855 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512859 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512863 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512867 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512869 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512872 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512875 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512877 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512881 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512883 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512886 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512888 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512891 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512894 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512896 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512899 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512901 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 08:20:29.522804 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512904 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512907 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512909 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512912 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512915 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512919 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512923 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512927 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512931 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512937 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512941 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512944 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512947 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512949 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512952 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512956 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512959 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512962 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512965 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 08:20:29.523401 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512968 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512970 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512973 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512976 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512980 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512983 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512985 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512988 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512990 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512993 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512996 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.512998 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513001 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513003 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513006 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513008 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513011 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513026 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513029 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513032 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 08:20:29.523864 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513035 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513038 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513041 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513044 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.513046 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.513590 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.522204 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.522221 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.523029 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.523037 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523113 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523118 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523122 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523125 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523128 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 08:20:29.524366 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523131 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523134 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523138 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523143 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523146 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523150 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523153 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523155 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523159 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523161 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523166 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523169 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523172 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523175 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523179 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523183 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523186 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523189 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523192 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 08:20:29.524740 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523195 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523197 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523200 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523203 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523206 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523209 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523213 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523215 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523218 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523221 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523223 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523226 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523228 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523231 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523234 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523236 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523239 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523242 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523244 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523247 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 08:20:29.525212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523250 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523252 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523256 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523258 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523261 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523264 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523266 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523269 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523272 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523275 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523277 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523280 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523282 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523286 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523289 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523292 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523295 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523298 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523301 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523304 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 08:20:29.525722 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523306 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523309 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523312 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523314 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523317 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523319 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523323 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523327 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523330 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523333 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523336 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523339 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523341 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523344 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523347 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523350 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523353 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523356 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523359 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 08:20:29.526222 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523361 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523364 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523366 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523369 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523372 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523375 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523377 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523380 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523383 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523385 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523389 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523391 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523394 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523397 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523399 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 08:20:29.526675 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523402 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.523407 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523516 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523521 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523524 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523527 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523530 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523533 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523536 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523539 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523541 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523544 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523547 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523550 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523553 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 19 08:20:29.527057 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523555 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523558 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523561 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523564 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523566 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523569 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523573 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523577 2585 feature_gate.go:328] unrecognized feature gate: Example Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523581 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523584 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523588 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523591 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523595 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523598 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523601 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523604 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523606 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523609 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523612 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 19 08:20:29.527428 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523615 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523617 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523620 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523623 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523625 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523628 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523630 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523633 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523636 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523638 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523641 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523644 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523647 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523650 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523653 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523655 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523658 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523661 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523663 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523666 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 19 08:20:29.527897 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523669 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523671 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523674 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523677 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523679 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523683 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523685 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523688 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523691 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523693 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523696 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523699 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523701 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523704 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523707 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523709 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523712 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523714 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523717 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523719 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 19 08:20:29.528388 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523722 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523725 2585 feature_gate.go:328] unrecognized feature gate: Example2 Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523728 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523730 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523733 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523735 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523739 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523741 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523744 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523747 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523749 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523752 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523755 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523757 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523760 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523763 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523765 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523769 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523773 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523775 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 19 08:20:29.528871 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523778 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523780 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523783 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523786 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523788 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523791 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:29.523793 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.523798 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.523897 2585 server.go:962] "Client rotation is on, will bootstrap in background" Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.527282 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.528202 2585 server.go:1019] "Starting client certificate rotation" Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.528300 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 19 08:20:29.529433 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.528329 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 19 08:20:29.530742 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.530728 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 19 08:20:29.532554 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.532538 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 19 08:20:29.540632 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.540613 2585 log.go:25] "Validated CRI v1 runtime API" Sep 19 08:20:29.542311 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.542296 2585 log.go:25] "Validated CRI v1 image API" Sep 19 08:20:29.544553 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.544538 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 19 08:20:29.548942 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.548921 2585 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 99e06ca4-bc84-48cf-a1b3-c1b3673ece21:/dev/nvme0n1p3 a21e1e95-86c6-4c6d-9a34-ee90bd5b18a6:/dev/nvme0n1p4] Sep 19 08:20:29.549008 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.548942 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 19 08:20:29.555738 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.555607 2585 manager.go:216] Machine: {Timestamp:2026-09-19 08:20:29.554223077 +0000 UTC m=+0.325562643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101994 MemoryCapacity:33164472320 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d75c7dc9daec1f83179948f2f8e14 SystemUUID:ec2d75c7-dc9d-aec1-f831-79948f2f8e14 BootID:9e4f0e15-ad5d-4b28-8e9b-cd8767ea5182 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582234112 Type:vfs Inodes:4048397 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632894464 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:ee:fa:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:ee:fa:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:63:70:b0:9f:69 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164472320 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 19 08:20:29.555738 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.555727 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 19 08:20:29.555904 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.555850 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 19 08:20:29.557744 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.557715 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 19 08:20:29.557901 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.557745 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-127.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 19 08:20:29.558599 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.558553 2585 topology_manager.go:138] "Creating topology manager with none policy" Sep 19 08:20:29.558705 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.558603 2585 container_manager_linux.go:309] "Creating device plugin manager" Sep 19 08:20:29.559136 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.558669 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 19 08:20:29.559212 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.559157 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 19 08:20:29.559855 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.559830 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 19 08:20:29.560415 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.560399 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 19 08:20:29.564116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564105 2585 state_mem.go:36] "Initialized new in-memory state store" Sep 19 08:20:29.564247 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564237 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 19 08:20:29.564283 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564267 2585 kubelet.go:486] "Attempting to sync node with API server" Sep 19 08:20:29.564283 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564277 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 19 08:20:29.564338 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564289 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 19 08:20:29.564338 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564298 2585 kubelet.go:398] "Adding apiserver pod source" Sep 19 08:20:29.564338 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564312 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 19 08:20:29.564677 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564665 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 19 08:20:29.564713 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.564698 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 19 08:20:29.566738 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.566721 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1" Sep 19 08:20:29.568801 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.568788 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 19 08:20:29.568854 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.568818 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 19 08:20:29.569866 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569854 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 19 08:20:29.569921 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569882 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 19 08:20:29.569921 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569891 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 19 08:20:29.569921 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569896 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 19 08:20:29.569921 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569906 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 19 08:20:29.569921 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569913 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 19 08:20:29.570067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569931 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 19 08:20:29.570067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569938 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 19 08:20:29.570067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569946 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 19 08:20:29.570067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569951 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 19 08:20:29.570067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569960 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 19 08:20:29.570067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.569965 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 19 08:20:29.570688 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.570679 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 19 08:20:29.570688 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.570688 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 19 08:20:29.573540 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.573518 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 19 08:20:29.573627 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.573528 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-127.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 19 08:20:29.573627 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.573534 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 19 08:20:29.574380 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.574367 2585 server.go:1268] "Started kubelet" Sep 19 08:20:29.574476 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.574448 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 19 08:20:29.574569 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.574471 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 19 08:20:29.574618 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.574602 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 19 08:20:29.575358 ip-10-0-139-127 systemd[1]: Started Kubernetes Kubelet. Sep 19 08:20:29.575701 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.575685 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 19 08:20:29.577254 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.577090 2585 server.go:310] "Adding debug handlers to kubelet server" Sep 19 08:20:29.580317 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.580297 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 19 08:20:29.583582 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.583564 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 19 08:20:29.583993 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.583009 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-127.ec2.internal.18d6ab19e1a18599 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-127.ec2.internal,UID:ip-10-0-139-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-127.ec2.internal,},FirstTimestamp:2026-09-19 08:20:29.574342041 +0000 UTC m=+0.345681605,LastTimestamp:2026-09-19 08:20:29.574342041 +0000 UTC m=+0.345681605,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-127.ec2.internal,}" Sep 19 08:20:29.584108 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.584072 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 19 08:20:29.584217 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.584196 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:29.584778 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.584747 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 19 08:20:29.584778 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.584774 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 19 08:20:29.584911 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.584751 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 19 08:20:29.584911 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.584881 2585 reconstruct.go:97] "Volume reconstruction finished" Sep 19 08:20:29.584911 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.584887 2585 reconciler.go:29] "Reconciler: start to sync state" Sep 19 08:20:29.585060 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.584967 2585 factory.go:55] Registering systemd factory Sep 19 08:20:29.585060 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.584984 2585 factory.go:223] Registration of the systemd container factory successfully Sep 19 08:20:29.585193 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.585178 2585 factory.go:153] Registering CRI-O factory Sep 19 08:20:29.585193 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.585195 2585 factory.go:223] Registration of the crio container factory successfully Sep 19 08:20:29.585337 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.585241 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 19 08:20:29.585337 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.585260 2585 factory.go:103] Registering Raw factory Sep 19 08:20:29.585337 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.585275 2585 manager.go:1196] Started watching for new ooms in manager Sep 19 08:20:29.586202 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.586184 2585 manager.go:349] Starting recovery of all containers Sep 19 08:20:29.586431 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.586404 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-127.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 19 08:20:29.586539 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.586507 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 19 08:20:29.586833 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.586813 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n5fwp" Sep 19 08:20:29.593657 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.593635 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n5fwp" Sep 19 08:20:29.598946 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.598930 2585 manager.go:354] Recovery completed Sep 19 08:20:29.602845 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.602829 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 08:20:29.605723 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.605704 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientMemory" Sep 19 08:20:29.605799 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.605739 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 08:20:29.605799 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.605755 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientPID" Sep 19 08:20:29.606469 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.606453 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 19 08:20:29.606469 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.606467 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 19 08:20:29.606572 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.606459 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 08:20:29.606572 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.606484 2585 state_mem.go:36] "Initialized new in-memory state store" Sep 19 08:20:29.607679 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.607655 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientMemory" Sep 19 08:20:29.607757 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.607684 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 08:20:29.607757 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.607694 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientPID" Sep 19 08:20:29.607821 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.607680 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-127.ec2.internal.18d6ab19e3805af2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-127.ec2.internal,UID:ip-10-0-139-127.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-127.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-127.ec2.internal,},FirstTimestamp:2026-09-19 08:20:29.605722866 +0000 UTC m=+0.377062431,LastTimestamp:2026-09-19 08:20:29.605722866 +0000 UTC m=+0.377062431,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-127.ec2.internal,}" Sep 19 08:20:29.609156 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.609142 2585 policy_none.go:49] "None policy: Start" Sep 19 08:20:29.609214 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.609162 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 19 08:20:29.609214 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.609174 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 19 08:20:29.611657 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.611644 2585 policy_none.go:47] "Start" Sep 19 08:20:29.640756 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.640598 2585 manager.go:337] "Starting Device Plugin manager" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.640798 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.640814 2585 server.go:87] "Starting device plugin registration server" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.641102 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.641116 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.641206 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.641281 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.641290 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.641782 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 19 08:20:29.660849 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.641829 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:29.739459 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.739424 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 19 08:20:29.740643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.740624 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 19 08:20:29.740742 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.740676 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 19 08:20:29.740742 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.740700 2585 kubelet.go:2448] "Starting kubelet main sync loop" Sep 19 08:20:29.740833 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.740794 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 19 08:20:29.741266 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.741247 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 08:20:29.742189 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.742168 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientMemory" Sep 19 08:20:29.742262 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.742206 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 08:20:29.742262 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.742221 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientPID" Sep 19 08:20:29.742262 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.742247 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.743044 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.743010 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 08:20:29.750165 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.750149 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.750251 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.750171 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-127.ec2.internal\": node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:29.764311 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.764246 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:29.841922 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.841871 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal"] Sep 19 08:20:29.842057 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.841967 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 08:20:29.842893 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.842879 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientMemory" Sep 19 08:20:29.842941 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.842909 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 08:20:29.842941 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.842919 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientPID" Sep 19 08:20:29.845234 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.845223 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 08:20:29.845346 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.845330 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.845444 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.845356 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 08:20:29.846922 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.846888 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientMemory" Sep 19 08:20:29.846922 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.846908 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientMemory" Sep 19 08:20:29.846922 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.846919 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 08:20:29.847100 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.846930 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 08:20:29.847100 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.846930 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientPID" Sep 19 08:20:29.847100 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.846941 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientPID" Sep 19 08:20:29.849121 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.849108 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.849168 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.849132 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 19 08:20:29.849803 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.849790 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientMemory" Sep 19 08:20:29.849873 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.849812 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasNoDiskPressure" Sep 19 08:20:29.849873 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.849822 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeHasSufficientPID" Sep 19 08:20:29.864456 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.864438 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:29.877715 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.877699 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-127.ec2.internal\" not found" node="ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.881971 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.881957 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-127.ec2.internal\" not found" node="ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.885467 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.885451 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5e2461d28fe7d2289f97533a4da81dc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-127.ec2.internal\" (UID: \"a5e2461d28fe7d2289f97533a4da81dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.885524 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.885474 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d536fa28c7445c394751037b6f939e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal\" (UID: \"a5d536fa28c7445c394751037b6f939e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.885524 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.885491 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d536fa28c7445c394751037b6f939e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal\" (UID: \"a5d536fa28c7445c394751037b6f939e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.964957 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:29.964938 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:29.986227 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.986205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d536fa28c7445c394751037b6f939e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal\" (UID: \"a5d536fa28c7445c394751037b6f939e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.986342 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.986232 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d536fa28c7445c394751037b6f939e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal\" (UID: \"a5d536fa28c7445c394751037b6f939e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.986342 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.986248 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5e2461d28fe7d2289f97533a4da81dc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-127.ec2.internal\" (UID: \"a5e2461d28fe7d2289f97533a4da81dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.986342 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.986278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5e2461d28fe7d2289f97533a4da81dc-config\") pod \"kube-apiserver-proxy-ip-10-0-139-127.ec2.internal\" (UID: \"a5e2461d28fe7d2289f97533a4da81dc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.986342 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.986288 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a5d536fa28c7445c394751037b6f939e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal\" (UID: \"a5d536fa28c7445c394751037b6f939e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:29.986342 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:29.986279 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a5d536fa28c7445c394751037b6f939e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal\" (UID: \"a5d536fa28c7445c394751037b6f939e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:30.065531 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:30.065467 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:30.165968 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:30.165934 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:30.182110 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:30.182079 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:30.185253 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:30.185235 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" Sep 19 08:20:31.826172 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:31.826145 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:31.926710 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:31.926652 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:32.027220 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:32.027191 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:32.087718 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.087687 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 19 08:20:32.088220 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.087833 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 19 08:20:32.128168 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:32.128141 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:32.159155 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.159121 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-18 08:15:31 +0000 UTC" deadline="2028-05-31 09:09:23.317168908 +0000 UTC" Sep 19 08:20:32.159155 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.159146 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14880h48m51.158025687s" Sep 19 08:20:32.228504 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:32.228432 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:32.230871 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.230846 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 08:20:32.294490 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:32.294457 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d536fa28c7445c394751037b6f939e.slice/crio-c2aefe992233728244e5c9c0dcb596b98a8f8fa1419d34d87549293793a59d68 WatchSource:0}: Error finding container c2aefe992233728244e5c9c0dcb596b98a8f8fa1419d34d87549293793a59d68: Status 404 returned error can't find the container with id c2aefe992233728244e5c9c0dcb596b98a8f8fa1419d34d87549293793a59d68 Sep 19 08:20:32.294824 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:32.294796 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5e2461d28fe7d2289f97533a4da81dc.slice/crio-4b1b0cbcb6f1797400eb1c3c33f61e20b4186ffd4659fb977692b7c36e776283 WatchSource:0}: Error finding container 4b1b0cbcb6f1797400eb1c3c33f61e20b4186ffd4659fb977692b7c36e776283: Status 404 returned error can't find the container with id 4b1b0cbcb6f1797400eb1c3c33f61e20b4186ffd4659fb977692b7c36e776283 Sep 19 08:20:32.299644 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.299624 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 08:20:32.303368 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.303330 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" event={"ID":"a5d536fa28c7445c394751037b6f939e","Type":"ContainerStarted","Data":"c2aefe992233728244e5c9c0dcb596b98a8f8fa1419d34d87549293793a59d68"} Sep 19 08:20:32.304403 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.304379 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" event={"ID":"a5e2461d28fe7d2289f97533a4da81dc","Type":"ContainerStarted","Data":"4b1b0cbcb6f1797400eb1c3c33f61e20b4186ffd4659fb977692b7c36e776283"} Sep 19 08:20:32.329270 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:32.329246 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:32.429711 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:32.429689 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:32.530283 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:32.530217 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-127.ec2.internal\" not found" Sep 19 08:20:32.549956 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.549933 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 08:20:32.626087 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.626057 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 08:20:32.644305 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.644282 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" Sep 19 08:20:32.655952 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.655930 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 19 08:20:32.656905 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.656893 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" Sep 19 08:20:32.664892 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:32.664875 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 19 08:20:33.125935 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.125907 2585 apiserver.go:52] "Watching apiserver" Sep 19 08:20:33.133338 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.133313 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 19 08:20:33.133921 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.133892 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-l9799","openshift-ovn-kubernetes/ovnkube-node-czp6j","kube-system/konnectivity-agent-vzdf4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k","openshift-cluster-node-tuning-operator/tuned-6nvjv","openshift-image-registry/node-ca-mvk4d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal","openshift-multus/multus-additional-cni-plugins-rwwtw","openshift-multus/multus-ndz5k","openshift-network-operator/iptables-alerter-jhxfp","kube-system/global-pull-secret-syncer-mz86w","kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal","openshift-dns/node-resolver-9c7kk","openshift-multus/network-metrics-daemon-7bd8s"] Sep 19 08:20:33.136425 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.136372 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.138764 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.138739 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.139056 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.139036 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 19 08:20:33.139323 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.139304 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-djbpz\"" Sep 19 08:20:33.139595 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.139579 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 19 08:20:33.139720 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.139697 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 19 08:20:33.140705 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.140686 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 19 08:20:33.141487 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.141160 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 19 08:20:33.141487 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.141464 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mtzfs\"" Sep 19 08:20:33.142101 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.141789 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:33.143721 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.143702 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 19 08:20:33.144070 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.144051 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 19 08:20:33.144166 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.144147 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 19 08:20:33.144276 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.144261 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-svqc2\"" Sep 19 08:20:33.144374 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.144350 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.146556 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.146512 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 19 08:20:33.146819 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.146647 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 19 08:20:33.146908 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.146710 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.147500 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.147482 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 19 08:20:33.147745 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.147728 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gwm5c\"" Sep 19 08:20:33.149359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.149045 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 19 08:20:33.149359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.149073 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 19 08:20:33.149359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.149100 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.149359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.149155 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8dkdg\"" Sep 19 08:20:33.151488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.151246 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 19 08:20:33.151488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.151362 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.151488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.151464 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 19 08:20:33.152187 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.151833 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kgl4c\"" Sep 19 08:20:33.152187 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.152167 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 19 08:20:33.152982 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.152855 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysctl-d\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153392 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.152900 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n55xz\" (UniqueName: \"kubernetes.io/projected/8f8fee67-86f5-4706-a9f3-d368cece2657-kube-api-access-n55xz\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.153392 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-sys\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153392 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153292 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-lib-modules\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153392 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153327 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5gqg\" (UniqueName: \"kubernetes.io/projected/7232a07f-d78f-4e05-b33d-9e28c67839e7-kube-api-access-p5gqg\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.153643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153403 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w947p\" (UniqueName: \"kubernetes.io/projected/ff2f92e7-7821-4a5c-9807-e81088d7e25f-kube-api-access-w947p\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.153643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153443 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-kubernetes\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-run\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153573 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-tuned\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153611 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-socket-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.153813 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153644 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-host\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153813 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2f92e7-7821-4a5c-9807-e81088d7e25f-hosts-file\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.153813 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7-konnectivity-ca\") pod \"konnectivity-agent-vzdf4\" (UID: \"6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7\") " pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:33.153813 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153742 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-registration-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.153813 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153771 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-systemd\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153969 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153814 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prxnh\" (UniqueName: \"kubernetes.io/projected/40a50508-3837-4eea-b588-e34cc16aa8cb-kube-api-access-prxnh\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153969 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153845 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysctl-conf\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153969 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153869 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-var-lib-kubelet\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153969 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153895 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40a50508-3837-4eea-b588-e34cc16aa8cb-tmp\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.153969 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153922 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2f92e7-7821-4a5c-9807-e81088d7e25f-tmp-dir\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.154116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.153986 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.154116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154016 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysconfig\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.154116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154041 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7232a07f-d78f-4e05-b33d-9e28c67839e7-iptables-alerter-script\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.154116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154068 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7232a07f-d78f-4e05-b33d-9e28c67839e7-host-slash\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.154116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154096 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.154373 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154123 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-device-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.154373 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154154 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-sys-fs\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.154373 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154179 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-modprobe-d\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.154373 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154207 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7-agent-certs\") pod \"konnectivity-agent-vzdf4\" (UID: \"6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7\") " pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:33.154586 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154508 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 19 08:20:33.155294 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154682 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vfnl4\"" Sep 19 08:20:33.155294 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154793 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 19 08:20:33.155294 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.154966 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 19 08:20:33.155294 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.155075 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 19 08:20:33.155294 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.155230 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 19 08:20:33.157015 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.156379 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:33.157015 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.156466 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:33.159755 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.159719 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.161896 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.161873 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 19 08:20:33.161983 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.161917 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 19 08:20:33.161983 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.161945 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 19 08:20:33.162208 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.162170 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:33.162289 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.162226 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:33.162289 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.162253 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 19 08:20:33.162388 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.162338 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 19 08:20:33.162388 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.162344 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jg9qf\"" Sep 19 08:20:33.162504 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.162485 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 19 08:20:33.164738 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.164717 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.164817 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.164785 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:33.167632 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.167260 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.169450 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.169429 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 19 08:20:33.169706 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.169688 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qlwpn\"" Sep 19 08:20:33.169795 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.169708 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 19 08:20:33.179589 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.179569 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fntb9" Sep 19 08:20:33.186868 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.186847 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fntb9" Sep 19 08:20:33.246264 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.246238 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 19 08:20:33.254465 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254424 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-run-netns\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.254465 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254463 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-systemd\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.254680 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254487 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-node-log\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.254680 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254553 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.254680 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254579 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6mvs\" (UniqueName: \"kubernetes.io/projected/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-kube-api-access-l6mvs\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.254680 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-systemd\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.254848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254703 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r579k\" (UniqueName: \"kubernetes.io/projected/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-kube-api-access-r579k\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:33.254848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254725 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-systemd\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.254848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254737 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-cni-netd\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.254848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254768 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-conf-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.254848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254797 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysctl-conf\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.254848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254837 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-var-lib-kubelet\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254865 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40a50508-3837-4eea-b588-e34cc16aa8cb-tmp\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254884 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-system-cni-dir\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254938 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysctl-conf\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254954 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254983 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysconfig\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.254984 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-var-lib-kubelet\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255018 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7232a07f-d78f-4e05-b33d-9e28c67839e7-iptables-alerter-script\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255082 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-device-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-sys-fs\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255158 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmpl\" (UniqueName: \"kubernetes.io/projected/492622cf-75d6-416d-bc84-1ba3effd3382-kube-api-access-cbmpl\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.255186 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255179 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-kubelet\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255199 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-slash\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255224 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysconfig\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255303 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-sys-fs\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-cni-bin\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255359 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-device-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255404 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-systemd-units\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255433 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255473 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-cnibin\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255494 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/beb9fb0e-3e64-42d6-a2b1-673a904852bb-cni-binary-copy\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-socket-dir-parent\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255510 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255563 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-hostroot\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255615 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khnj\" (UniqueName: \"kubernetes.io/projected/beb9fb0e-3e64-42d6-a2b1-673a904852bb-kube-api-access-9khnj\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255649 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91530dee-a144-497b-b63a-3b88b4878905-dbus\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.255704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysctl-d\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/492622cf-75d6-416d-bc84-1ba3effd3382-serviceca\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255751 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/7232a07f-d78f-4e05-b33d-9e28c67839e7-iptables-alerter-script\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255762 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovn-node-metrics-cert\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255810 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255832 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-cni-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255877 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-cni-multus\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255901 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255926 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-lib-modules\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.255897 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-sysctl-d\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256001 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w947p\" (UniqueName: \"kubernetes.io/projected/ff2f92e7-7821-4a5c-9807-e81088d7e25f-kube-api-access-w947p\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256047 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovnkube-config\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256071 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovnkube-script-lib\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256112 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cni-binary-copy\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256137 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-lib-modules\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256137 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-k8s-cni-cncf-io\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256189 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91530dee-a144-497b-b63a-3b88b4878905-kubelet-config\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.256488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256265 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-kubernetes\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-run\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256320 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-socket-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256353 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-run\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256365 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-kubernetes\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256345 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-etc-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256410 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-env-overrides\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-kubelet\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256477 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-socket-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256542 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7-konnectivity-ca\") pod \"konnectivity-agent-vzdf4\" (UID: \"6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7\") " pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/492622cf-75d6-416d-bc84-1ba3effd3382-host\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256607 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prxnh\" (UniqueName: \"kubernetes.io/projected/40a50508-3837-4eea-b588-e34cc16aa8cb-kube-api-access-prxnh\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256674 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4472\" (UniqueName: \"kubernetes.io/projected/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-kube-api-access-m4472\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256695 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2f92e7-7821-4a5c-9807-e81088d7e25f-tmp-dir\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256716 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256747 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-run-ovn-kubernetes\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.257371 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256769 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-system-cni-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256799 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-multus-certs\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256837 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7232a07f-d78f-4e05-b33d-9e28c67839e7-host-slash\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256865 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-var-lib-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256944 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.256990 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-daemon-config\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-modprobe-d\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257073 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7-agent-certs\") pod \"konnectivity-agent-vzdf4\" (UID: \"6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7\") " pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257134 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7-konnectivity-ca\") pod \"konnectivity-agent-vzdf4\" (UID: \"6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7\") " pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7232a07f-d78f-4e05-b33d-9e28c67839e7-host-slash\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257330 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-modprobe-d\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257350 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-etc-selinux\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257643 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-ovn\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257672 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff2f92e7-7821-4a5c-9807-e81088d7e25f-tmp-dir\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257677 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n55xz\" (UniqueName: \"kubernetes.io/projected/8f8fee67-86f5-4706-a9f3-d368cece2657-kube-api-access-n55xz\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.258164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257731 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cnibin\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257772 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-etc-kubernetes\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257802 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-sys\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257877 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-sys\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257879 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5gqg\" (UniqueName: \"kubernetes.io/projected/7232a07f-d78f-4e05-b33d-9e28c67839e7-kube-api-access-p5gqg\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.257998 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-tuned\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258038 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-log-socket\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258081 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-netns\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258139 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-cni-bin\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258171 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-host\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258198 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2f92e7-7821-4a5c-9807-e81088d7e25f-hosts-file\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-registration-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff2f92e7-7821-4a5c-9807-e81088d7e25f-hosts-file\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258339 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8f8fee67-86f5-4706-a9f3-d368cece2657-registration-dir\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.259065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.258251 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/40a50508-3837-4eea-b588-e34cc16aa8cb-host\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.259781 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.259105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40a50508-3837-4eea-b588-e34cc16aa8cb-tmp\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.260512 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.260486 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7-agent-certs\") pod \"konnectivity-agent-vzdf4\" (UID: \"6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7\") " pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:33.261111 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.261062 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/40a50508-3837-4eea-b588-e34cc16aa8cb-etc-tuned\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.264560 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.264405 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w947p\" (UniqueName: \"kubernetes.io/projected/ff2f92e7-7821-4a5c-9807-e81088d7e25f-kube-api-access-w947p\") pod \"node-resolver-9c7kk\" (UID: \"ff2f92e7-7821-4a5c-9807-e81088d7e25f\") " pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.265129 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.265087 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5gqg\" (UniqueName: \"kubernetes.io/projected/7232a07f-d78f-4e05-b33d-9e28c67839e7-kube-api-access-p5gqg\") pod \"iptables-alerter-jhxfp\" (UID: \"7232a07f-d78f-4e05-b33d-9e28c67839e7\") " pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.265280 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.265260 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prxnh\" (UniqueName: \"kubernetes.io/projected/40a50508-3837-4eea-b588-e34cc16aa8cb-kube-api-access-prxnh\") pod \"tuned-6nvjv\" (UID: \"40a50508-3837-4eea-b588-e34cc16aa8cb\") " pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.265535 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.265503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n55xz\" (UniqueName: \"kubernetes.io/projected/8f8fee67-86f5-4706-a9f3-d368cece2657-kube-api-access-n55xz\") pod \"aws-ebs-csi-driver-node-h4n5k\" (UID: \"8f8fee67-86f5-4706-a9f3-d368cece2657\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.358750 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358715 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-etc-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.358750 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358753 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-env-overrides\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358774 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-kubelet\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358796 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/492622cf-75d6-416d-bc84-1ba3effd3382-host\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358819 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358841 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4472\" (UniqueName: \"kubernetes.io/projected/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-kube-api-access-m4472\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358841 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-kubelet\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358865 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-run-ovn-kubernetes\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358868 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/492622cf-75d6-416d-bc84-1ba3effd3382-host\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358889 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-system-cni-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358899 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-etc-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358927 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-run-ovn-kubernetes\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358909 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-multus-certs\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358960 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-multus-certs\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358969 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-system-cni-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.358963 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:33.358974 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.358981 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-var-lib-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359001 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359017 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-daemon-config\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.359074 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs podName:30dd1f4f-1ad8-4cca-94f9-ae57d1e43449 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:33.859032856 +0000 UTC m=+3.070866695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs") pod "network-metrics-daemon-7bd8s" (UID: "30dd1f4f-1ad8-4cca-94f9-ae57d1e43449") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359095 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-ovn\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-var-lib-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359124 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cnibin\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359159 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cnibin\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359164 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-etc-kubernetes\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359187 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359198 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359183 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-ovn\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359211 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-etc-kubernetes\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359228 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-log-socket\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359231 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-env-overrides\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359252 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-openvswitch\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359290 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-netns\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.359613 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-log-socket\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359352 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-cni-bin\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359368 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-netns\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359411 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-cni-bin\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359439 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-run-netns\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359462 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-systemd\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359482 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-node-log\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359502 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359507 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-run-netns\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359539 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6mvs\" (UniqueName: \"kubernetes.io/projected/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-kube-api-access-l6mvs\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359559 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r579k\" (UniqueName: \"kubernetes.io/projected/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-kube-api-access-r579k\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359572 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-node-log\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-cni-netd\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359589 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359596 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-cni-netd\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359609 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-conf-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-run-systemd\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.360274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359637 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-system-cni-dir\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359667 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-conf-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359671 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-system-cni-dir\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359560 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-daemon-config\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmpl\" (UniqueName: \"kubernetes.io/projected/492622cf-75d6-416d-bc84-1ba3effd3382-kube-api-access-cbmpl\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359814 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-kubelet\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359843 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-slash\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359855 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-kubelet\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359867 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-cni-bin\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359891 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-systemd-units\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359895 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-slash\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359918 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-host-cni-bin\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359920 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359942 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-systemd-units\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359958 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-cnibin\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359986 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-cnibin\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.359995 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/beb9fb0e-3e64-42d6-a2b1-673a904852bb-cni-binary-copy\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360028 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-socket-dir-parent\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-hostroot\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360075 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khnj\" (UniqueName: \"kubernetes.io/projected/beb9fb0e-3e64-42d6-a2b1-673a904852bb-kube-api-access-9khnj\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360097 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91530dee-a144-497b-b63a-3b88b4878905-dbus\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360120 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/492622cf-75d6-416d-bc84-1ba3effd3382-serviceca\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360141 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovn-node-metrics-cert\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360166 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360194 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-cni-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360217 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-cni-multus\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360241 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360268 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovnkube-config\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360291 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovnkube-script-lib\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360315 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cni-binary-copy\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360339 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-k8s-cni-cncf-io\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91530dee-a144-497b-b63a-3b88b4878905-kubelet-config\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360422 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360433 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/91530dee-a144-497b-b63a-3b88b4878905-dbus\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.361768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360389 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-cni-dir\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-multus-socket-dir-parent\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360466 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/beb9fb0e-3e64-42d6-a2b1-673a904852bb-cni-binary-copy\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.360538 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.360595 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret podName:91530dee-a144-497b-b63a-3b88b4878905 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:33.86058081 +0000 UTC m=+3.072414662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret") pod "global-pull-secret-syncer-mz86w" (UID: "91530dee-a144-497b-b63a-3b88b4878905") : object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360641 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-run-k8s-cni-cncf-io\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360339 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-hostroot\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360701 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/91530dee-a144-497b-b63a-3b88b4878905-kubelet-config\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360865 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/492622cf-75d6-416d-bc84-1ba3effd3382-serviceca\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360913 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/beb9fb0e-3e64-42d6-a2b1-673a904852bb-host-var-lib-cni-multus\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.360937 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cni-binary-copy\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.361117 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.361275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovnkube-config\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.362281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.361339 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovnkube-script-lib\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.363098 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.363078 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-ovn-node-metrics-cert\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.366398 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.366377 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 08:20:33.366499 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.366411 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 08:20:33.366499 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.366425 2585 projected.go:196] Error preparing data for projected volume kube-api-access-kpbkq for pod openshift-network-diagnostics/network-check-target-l9799: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:33.366628 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.366509 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq podName:aa486a4a-823d-40ca-b1d1-df39a7a1baf8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:33.866492356 +0000 UTC m=+3.078326199 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kpbkq" (UniqueName: "kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq") pod "network-check-target-l9799" (UID: "aa486a4a-823d-40ca-b1d1-df39a7a1baf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:33.368626 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.368605 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4472\" (UniqueName: \"kubernetes.io/projected/c0830ee0-b0ea-4e9d-bac7-3f61cc93712e-kube-api-access-m4472\") pod \"ovnkube-node-czp6j\" (UID: \"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e\") " pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.368951 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.368932 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmpl\" (UniqueName: \"kubernetes.io/projected/492622cf-75d6-416d-bc84-1ba3effd3382-kube-api-access-cbmpl\") pod \"node-ca-mvk4d\" (UID: \"492622cf-75d6-416d-bc84-1ba3effd3382\") " pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.369135 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.369118 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r579k\" (UniqueName: \"kubernetes.io/projected/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-kube-api-access-r579k\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:33.369390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.369359 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6mvs\" (UniqueName: \"kubernetes.io/projected/cacb029c-d5ba-4bdf-b8ae-cbd5b809de91-kube-api-access-l6mvs\") pod \"multus-additional-cni-plugins-rwwtw\" (UID: \"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91\") " pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.369695 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.369674 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khnj\" (UniqueName: \"kubernetes.io/projected/beb9fb0e-3e64-42d6-a2b1-673a904852bb-kube-api-access-9khnj\") pod \"multus-ndz5k\" (UID: \"beb9fb0e-3e64-42d6-a2b1-673a904852bb\") " pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.371917 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.371897 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 19 08:20:33.450962 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.450881 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jhxfp" Sep 19 08:20:33.460233 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.460210 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:33.469032 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.469011 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9c7kk" Sep 19 08:20:33.476643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.476623 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" Sep 19 08:20:33.484571 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.484551 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" Sep 19 08:20:33.492141 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.492121 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mvk4d" Sep 19 08:20:33.499714 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.499695 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ndz5k" Sep 19 08:20:33.506321 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.506299 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:33.511894 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.511865 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" Sep 19 08:20:33.863607 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.863530 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:33.863607 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.863566 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:33.863811 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.863675 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:33.863811 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.863679 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:33.863811 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.863727 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs podName:30dd1f4f-1ad8-4cca-94f9-ae57d1e43449 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:34.863712331 +0000 UTC m=+4.075546171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs") pod "network-metrics-daemon-7bd8s" (UID: "30dd1f4f-1ad8-4cca-94f9-ae57d1e43449") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:33.863811 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.863741 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret podName:91530dee-a144-497b-b63a-3b88b4878905 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:34.863734951 +0000 UTC m=+4.075568786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret") pod "global-pull-secret-syncer-mz86w" (UID: "91530dee-a144-497b-b63a-3b88b4878905") : object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:33.964040 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:33.964011 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:33.964186 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.964150 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 08:20:33.964186 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.964166 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 08:20:33.964186 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.964175 2585 projected.go:196] Error preparing data for projected volume kube-api-access-kpbkq for pod openshift-network-diagnostics/network-check-target-l9799: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:33.964338 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:33.964230 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq podName:aa486a4a-823d-40ca-b1d1-df39a7a1baf8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:34.964210941 +0000 UTC m=+4.176044776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpbkq" (UniqueName: "kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq") pod "network-check-target-l9799" (UID: "aa486a4a-823d-40ca-b1d1-df39a7a1baf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:34.190076 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.189991 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:33 +0000 UTC" deadline="2028-06-23 23:11:53.970250586 +0000 UTC" Sep 19 08:20:34.190076 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.190030 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15446h51m19.780227921s" Sep 19 08:20:34.230050 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:34.230023 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcacb029c_d5ba_4bdf_b8ae_cbd5b809de91.slice/crio-a1b40be6b1889cd8a3581ef8237088ed9f0b42a672473944df952587d8f3fa17 WatchSource:0}: Error finding container a1b40be6b1889cd8a3581ef8237088ed9f0b42a672473944df952587d8f3fa17: Status 404 returned error can't find the container with id a1b40be6b1889cd8a3581ef8237088ed9f0b42a672473944df952587d8f3fa17 Sep 19 08:20:34.250402 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:34.250365 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb9fb0e_3e64_42d6_a2b1_673a904852bb.slice/crio-9fc91aaab9783735e5b862978efaf45582a7809314c288b943920833656e9e6b WatchSource:0}: Error finding container 9fc91aaab9783735e5b862978efaf45582a7809314c288b943920833656e9e6b: Status 404 returned error can't find the container with id 9fc91aaab9783735e5b862978efaf45582a7809314c288b943920833656e9e6b Sep 19 08:20:34.250963 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:34.250938 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8fee67_86f5_4706_a9f3_d368cece2657.slice/crio-91366eba4a720241ce0547a609c1ad166d72a8f5368c8c5bfb20cbb5486c0ce6 WatchSource:0}: Error finding container 91366eba4a720241ce0547a609c1ad166d72a8f5368c8c5bfb20cbb5486c0ce6: Status 404 returned error can't find the container with id 91366eba4a720241ce0547a609c1ad166d72a8f5368c8c5bfb20cbb5486c0ce6 Sep 19 08:20:34.256473 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:34.256453 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7232a07f_d78f_4e05_b33d_9e28c67839e7.slice/crio-820bd770f2a807c43bb1bb6888824ada1965d3c5d512c630bd8f16bafce0bd23 WatchSource:0}: Error finding container 820bd770f2a807c43bb1bb6888824ada1965d3c5d512c630bd8f16bafce0bd23: Status 404 returned error can't find the container with id 820bd770f2a807c43bb1bb6888824ada1965d3c5d512c630bd8f16bafce0bd23 Sep 19 08:20:34.257291 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:34.257267 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2f92e7_7821_4a5c_9807_e81088d7e25f.slice/crio-f0a7d16293f1dcd766fb2eb121d22f21a16f21e54498409f98bcd6ce5f3dda86 WatchSource:0}: Error finding container f0a7d16293f1dcd766fb2eb121d22f21a16f21e54498409f98bcd6ce5f3dda86: Status 404 returned error can't find the container with id f0a7d16293f1dcd766fb2eb121d22f21a16f21e54498409f98bcd6ce5f3dda86 Sep 19 08:20:34.258136 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:34.258067 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6454de79_4ae5_4e2b_a18d_ac6c4f32c7a7.slice/crio-fbd9482342478a93eb2367face557ba9ce6253788c79e9246f5451982512fda7 WatchSource:0}: Error finding container fbd9482342478a93eb2367face557ba9ce6253788c79e9246f5451982512fda7: Status 404 returned error can't find the container with id fbd9482342478a93eb2367face557ba9ce6253788c79e9246f5451982512fda7 Sep 19 08:20:34.259175 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:20:34.259155 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492622cf_75d6_416d_bc84_1ba3effd3382.slice/crio-21383a9fa7d57b30f8dfaf0980c36ecc0e4df9153e6209733c7de54ca1f20514 WatchSource:0}: Error finding container 21383a9fa7d57b30f8dfaf0980c36ecc0e4df9153e6209733c7de54ca1f20514: Status 404 returned error can't find the container with id 21383a9fa7d57b30f8dfaf0980c36ecc0e4df9153e6209733c7de54ca1f20514 Sep 19 08:20:34.308192 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.308027 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"f21ae1b299bb75b3eec2614a2888bb97965a8961cca5da76ae0b3f26068b0d34"} Sep 19 08:20:34.309185 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.309159 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" event={"ID":"40a50508-3837-4eea-b588-e34cc16aa8cb","Type":"ContainerStarted","Data":"c7ca154607aa816f8cfafc5a1a4d3466cd617e8d8b54f3c28005aec6f2ef6c30"} Sep 19 08:20:34.310355 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.310321 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9c7kk" event={"ID":"ff2f92e7-7821-4a5c-9807-e81088d7e25f","Type":"ContainerStarted","Data":"f0a7d16293f1dcd766fb2eb121d22f21a16f21e54498409f98bcd6ce5f3dda86"} Sep 19 08:20:34.311508 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.311483 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" event={"ID":"8f8fee67-86f5-4706-a9f3-d368cece2657","Type":"ContainerStarted","Data":"91366eba4a720241ce0547a609c1ad166d72a8f5368c8c5bfb20cbb5486c0ce6"} Sep 19 08:20:34.312534 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.312504 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ndz5k" event={"ID":"beb9fb0e-3e64-42d6-a2b1-673a904852bb","Type":"ContainerStarted","Data":"9fc91aaab9783735e5b862978efaf45582a7809314c288b943920833656e9e6b"} Sep 19 08:20:34.313565 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.313541 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvk4d" event={"ID":"492622cf-75d6-416d-bc84-1ba3effd3382","Type":"ContainerStarted","Data":"21383a9fa7d57b30f8dfaf0980c36ecc0e4df9153e6209733c7de54ca1f20514"} Sep 19 08:20:34.314676 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.314658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vzdf4" event={"ID":"6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7","Type":"ContainerStarted","Data":"fbd9482342478a93eb2367face557ba9ce6253788c79e9246f5451982512fda7"} Sep 19 08:20:34.315903 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.315855 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhxfp" event={"ID":"7232a07f-d78f-4e05-b33d-9e28c67839e7","Type":"ContainerStarted","Data":"820bd770f2a807c43bb1bb6888824ada1965d3c5d512c630bd8f16bafce0bd23"} Sep 19 08:20:34.317116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.317097 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" event={"ID":"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91","Type":"ContainerStarted","Data":"a1b40be6b1889cd8a3581ef8237088ed9f0b42a672473944df952587d8f3fa17"} Sep 19 08:20:34.872100 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.872016 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:34.872100 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.872080 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:34.872316 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:34.872233 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:34.872316 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:34.872296 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs podName:30dd1f4f-1ad8-4cca-94f9-ae57d1e43449 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:36.872278327 +0000 UTC m=+6.084112169 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs") pod "network-metrics-daemon-7bd8s" (UID: "30dd1f4f-1ad8-4cca-94f9-ae57d1e43449") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:34.872723 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:34.872691 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:34.872816 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:34.872741 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret podName:91530dee-a144-497b-b63a-3b88b4878905 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:36.872727053 +0000 UTC m=+6.084560893 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret") pod "global-pull-secret-syncer-mz86w" (UID: "91530dee-a144-497b-b63a-3b88b4878905") : object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:34.974090 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:34.973387 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:34.974090 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:34.973607 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 08:20:34.974090 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:34.973630 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 08:20:34.974090 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:34.973676 2585 projected.go:196] Error preparing data for projected volume kube-api-access-kpbkq for pod openshift-network-diagnostics/network-check-target-l9799: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:34.974090 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:34.973735 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq podName:aa486a4a-823d-40ca-b1d1-df39a7a1baf8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:36.973717733 +0000 UTC m=+6.185551572 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpbkq" (UniqueName: "kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq") pod "network-check-target-l9799" (UID: "aa486a4a-823d-40ca-b1d1-df39a7a1baf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:35.190619 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:35.190467 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-18 08:15:33 +0000 UTC" deadline="2028-05-11 07:17:32.919233159 +0000 UTC" Sep 19 08:20:35.190619 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:35.190542 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14398h56m57.72869575s" Sep 19 08:20:35.304188 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:35.303239 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:35.304188 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:35.303382 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:35.304188 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:35.303774 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:35.304188 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:35.303862 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:35.304188 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:35.303931 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:35.304188 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:35.304006 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:35.344685 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:35.344650 2585 generic.go:356] "Generic (PLEG): container finished" podID="a5d536fa28c7445c394751037b6f939e" containerID="d78fdc7a7cd026741a422b56a129d626b0bf1262e41e71633603b25d1c997f16" exitCode=0 Sep 19 08:20:35.344940 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:35.344911 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" event={"ID":"a5d536fa28c7445c394751037b6f939e","Type":"ContainerDied","Data":"d78fdc7a7cd026741a422b56a129d626b0bf1262e41e71633603b25d1c997f16"} Sep 19 08:20:35.364111 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:35.364083 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" event={"ID":"a5e2461d28fe7d2289f97533a4da81dc","Type":"ContainerStarted","Data":"8913362cecfb91292f4063f945184bd93abeaa9f190319c71c16ec7d733a874c"} Sep 19 08:20:36.383664 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:36.383580 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" event={"ID":"a5d536fa28c7445c394751037b6f939e","Type":"ContainerStarted","Data":"d42cc8d1792e837b9821019654d0f38ea45566b27defc5a3f1109ee1e0bd00dc"} Sep 19 08:20:36.401091 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:36.400144 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-127.ec2.internal" podStartSLOduration=4.400122987 podStartE2EDuration="4.400122987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:35.377687669 +0000 UTC m=+4.589521528" watchObservedRunningTime="2026-09-19 08:20:36.400122987 +0000 UTC m=+5.611956846" Sep 19 08:20:36.888203 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:36.888163 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:36.888384 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:36.888227 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:36.888546 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:36.888442 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:36.888546 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:36.888511 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs podName:30dd1f4f-1ad8-4cca-94f9-ae57d1e43449 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:40.888491259 +0000 UTC m=+10.100325102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs") pod "network-metrics-daemon-7bd8s" (UID: "30dd1f4f-1ad8-4cca-94f9-ae57d1e43449") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:36.888546 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:36.888537 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:36.888735 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:36.888603 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret podName:91530dee-a144-497b-b63a-3b88b4878905 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:40.888585989 +0000 UTC m=+10.100419834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret") pod "global-pull-secret-syncer-mz86w" (UID: "91530dee-a144-497b-b63a-3b88b4878905") : object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:36.990097 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:36.989254 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:36.990097 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:36.989547 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 08:20:36.990097 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:36.989611 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 08:20:36.990097 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:36.989625 2585 projected.go:196] Error preparing data for projected volume kube-api-access-kpbkq for pod openshift-network-diagnostics/network-check-target-l9799: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:36.990097 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:36.989689 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq podName:aa486a4a-823d-40ca-b1d1-df39a7a1baf8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:40.989670317 +0000 UTC m=+10.201504175 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpbkq" (UniqueName: "kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq") pod "network-check-target-l9799" (UID: "aa486a4a-823d-40ca-b1d1-df39a7a1baf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:37.302327 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:37.302246 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:37.302474 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:37.302376 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:37.302731 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:37.302694 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:37.302823 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:37.302800 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:37.302880 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:37.302871 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:37.302978 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:37.302962 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:39.301205 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:39.300935 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:39.301205 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:39.300934 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:39.301205 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:39.301092 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:39.301205 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:39.301155 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:39.301205 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:39.301209 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:39.301915 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:39.301273 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:40.923382 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:40.922668 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:40.923382 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:40.922777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:40.923382 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:40.922948 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:40.923382 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:40.923038 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret podName:91530dee-a144-497b-b63a-3b88b4878905 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:48.923015956 +0000 UTC m=+18.134849798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret") pod "global-pull-secret-syncer-mz86w" (UID: "91530dee-a144-497b-b63a-3b88b4878905") : object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:40.923382 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:40.923336 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:40.924027 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:40.923403 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs podName:30dd1f4f-1ad8-4cca-94f9-ae57d1e43449 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:48.923383522 +0000 UTC m=+18.135217370 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs") pod "network-metrics-daemon-7bd8s" (UID: "30dd1f4f-1ad8-4cca-94f9-ae57d1e43449") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:41.024228 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:41.024096 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:41.024399 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:41.024279 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 08:20:41.024399 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:41.024297 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 08:20:41.024399 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:41.024310 2585 projected.go:196] Error preparing data for projected volume kube-api-access-kpbkq for pod openshift-network-diagnostics/network-check-target-l9799: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:41.024399 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:41.024368 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq podName:aa486a4a-823d-40ca-b1d1-df39a7a1baf8 nodeName:}" failed. No retries permitted until 2026-09-19 08:20:49.024351309 +0000 UTC m=+18.236185166 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpbkq" (UniqueName: "kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq") pod "network-check-target-l9799" (UID: "aa486a4a-823d-40ca-b1d1-df39a7a1baf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:41.302049 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:41.301891 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:41.302049 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:41.302008 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:41.302049 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:41.302021 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:41.302300 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:41.302140 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:41.302300 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:41.302207 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:41.302300 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:41.302278 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:43.301264 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:43.301230 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:43.301776 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:43.301272 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:43.301776 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:43.301230 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:43.301776 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:43.301398 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:43.301776 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:43.301456 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:43.301776 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:43.301556 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:45.300915 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:45.300875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:45.301352 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:45.300875 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:45.301352 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:45.301002 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:45.301352 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:45.301004 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:45.301352 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:45.301097 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:45.301352 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:45.301178 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:47.300748 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:47.300719 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:47.301228 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:47.300720 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:47.301228 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:47.300844 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:47.301228 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:47.300925 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:47.301228 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:47.300724 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:47.301228 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:47.301076 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:48.981835 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:48.981795 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:48.982324 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:48.981855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:48.982324 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:48.981962 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:48.982324 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:48.981968 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:48.982324 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:48.982024 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs podName:30dd1f4f-1ad8-4cca-94f9-ae57d1e43449 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:04.98200813 +0000 UTC m=+34.193841965 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs") pod "network-metrics-daemon-7bd8s" (UID: "30dd1f4f-1ad8-4cca-94f9-ae57d1e43449") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:20:48.982324 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:48.982039 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret podName:91530dee-a144-497b-b63a-3b88b4878905 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:04.982032836 +0000 UTC m=+34.193866670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret") pod "global-pull-secret-syncer-mz86w" (UID: "91530dee-a144-497b-b63a-3b88b4878905") : object "kube-system"/"original-pull-secret" not registered Sep 19 08:20:49.082958 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:49.082924 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:49.083133 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:49.083061 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 08:20:49.083133 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:49.083081 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 08:20:49.083133 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:49.083093 2585 projected.go:196] Error preparing data for projected volume kube-api-access-kpbkq for pod openshift-network-diagnostics/network-check-target-l9799: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:49.083274 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:49.083152 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq podName:aa486a4a-823d-40ca-b1d1-df39a7a1baf8 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:05.083134979 +0000 UTC m=+34.294968818 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpbkq" (UniqueName: "kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq") pod "network-check-target-l9799" (UID: "aa486a4a-823d-40ca-b1d1-df39a7a1baf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:20:49.300973 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:49.300893 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:49.301119 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:49.300893 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:49.301119 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:49.301042 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:49.301119 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:49.301095 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:49.301274 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:49.301184 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:49.301274 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:49.301262 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:51.301672 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:51.301635 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:51.302145 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:51.301748 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:51.302145 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:51.301782 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:51.302145 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:51.301875 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:51.302145 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:51.301915 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:51.302145 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:51.302002 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:52.419606 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.419208 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"e7b7b96527fe0c2a15bc9484d4edf6b0e35f8459fb82d9c904dc782c5f0baaa4"} Sep 19 08:20:52.420057 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.419715 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"3d6a069722b2fda81014e6194e8ac9d1aed961cd55201007975537d2060f6b66"} Sep 19 08:20:52.420057 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.419773 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"bc172d8035d4be823cd7ee42b27f00c1371beba3c33eeb73e6d1af3a5d3239b8"} Sep 19 08:20:52.420057 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.419814 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"1c6127c4c1b45dbfb9444114647dd24cfd5c797e22d385f30a4a66cece33c9ce"} Sep 19 08:20:52.420759 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.420738 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" event={"ID":"40a50508-3837-4eea-b588-e34cc16aa8cb","Type":"ContainerStarted","Data":"a8acc8a30a7fea059fb1c0df4115e7f7e3d01555c029f81b5caa5b671c2264c0"} Sep 19 08:20:52.422276 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.422250 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9c7kk" event={"ID":"ff2f92e7-7821-4a5c-9807-e81088d7e25f","Type":"ContainerStarted","Data":"c8d0f907bac674e08e592c06424077b294fa0243c81230981eda79c5e83d8e94"} Sep 19 08:20:52.423572 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.423549 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" event={"ID":"8f8fee67-86f5-4706-a9f3-d368cece2657","Type":"ContainerStarted","Data":"d618efd2adf19373bbddb79465d2c6cabeea67c439d067e54d6ff3be05663f52"} Sep 19 08:20:52.425042 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.424999 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ndz5k" event={"ID":"beb9fb0e-3e64-42d6-a2b1-673a904852bb","Type":"ContainerStarted","Data":"eacc67e42e388303daee6ec6a7f01b818753e0f15698260997477a762358825f"} Sep 19 08:20:52.426589 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.426566 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mvk4d" event={"ID":"492622cf-75d6-416d-bc84-1ba3effd3382","Type":"ContainerStarted","Data":"6e4d795954e069be9cea35a9127cedf1539e976af2890d2f0ae216c626da9bc9"} Sep 19 08:20:52.428269 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.428249 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vzdf4" event={"ID":"6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7","Type":"ContainerStarted","Data":"f4cd6d38d504600cc0225e9b9293feed6e2c5a7d9542dd4b7f40584216ac11e9"} Sep 19 08:20:52.429962 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.429938 2585 generic.go:356] "Generic (PLEG): container finished" podID="cacb029c-d5ba-4bdf-b8ae-cbd5b809de91" containerID="1984e8f939b26cc1944479e57e8a5803b6f84a1be23bf7729a30fc635d6c83fc" exitCode=0 Sep 19 08:20:52.430063 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.429973 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" event={"ID":"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91","Type":"ContainerDied","Data":"1984e8f939b26cc1944479e57e8a5803b6f84a1be23bf7729a30fc635d6c83fc"} Sep 19 08:20:52.442064 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.442025 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-127.ec2.internal" podStartSLOduration=20.442012372 podStartE2EDuration="20.442012372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:36.401057325 +0000 UTC m=+5.612891175" watchObservedRunningTime="2026-09-19 08:20:52.442012372 +0000 UTC m=+21.653846229" Sep 19 08:20:52.442703 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.442674 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6nvjv" podStartSLOduration=3.713015494 podStartE2EDuration="21.44266382s" totalImagesPullingTime="17.729648326s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:52.441861584 +0000 UTC m=+21.653695441" watchObservedRunningTime="2026-09-19 08:20:52.44266382 +0000 UTC m=+21.654497676" Sep 19 08:20:52.456166 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.456127 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mvk4d" podStartSLOduration=3.724897883 podStartE2EDuration="21.456110964s" totalImagesPullingTime="17.731213081s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:52.456099183 +0000 UTC m=+21.667933040" watchObservedRunningTime="2026-09-19 08:20:52.456110964 +0000 UTC m=+21.667944822" Sep 19 08:20:52.472305 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.472273 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ndz5k" podStartSLOduration=3.604693955 podStartE2EDuration="21.472264169s" totalImagesPullingTime="17.867570214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:52.472057173 +0000 UTC m=+21.683891030" watchObservedRunningTime="2026-09-19 08:20:52.472264169 +0000 UTC m=+21.684098026" Sep 19 08:20:52.515670 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.515632 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9c7kk" podStartSLOduration=3.781623388 podStartE2EDuration="21.515620032s" totalImagesPullingTime="17.733996644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:52.515264601 +0000 UTC m=+21.727098459" watchObservedRunningTime="2026-09-19 08:20:52.515620032 +0000 UTC m=+21.727453889" Sep 19 08:20:52.537884 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:52.537835 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vzdf4" podStartSLOduration=11.191771581 podStartE2EDuration="21.537821748s" totalImagesPullingTime="10.346050167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:52.537493666 +0000 UTC m=+21.749327522" watchObservedRunningTime="2026-09-19 08:20:52.537821748 +0000 UTC m=+21.749655604" Sep 19 08:20:53.300908 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.300874 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:53.301136 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.300882 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:53.301136 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.301029 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:53.301136 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:53.301028 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:53.301136 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:53.301119 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:53.301351 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:53.301194 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:53.429582 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.429550 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:53.430332 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.430185 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:53.436895 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.436867 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"83d1934ca05b576ab6935dd87076decd1f201abfe1180fce86703d9a8d3dacab"} Sep 19 08:20:53.437016 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.436950 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"6ea7f026e96fec205ad111a0cb927fb3923b114f7b299efa22ce68e86a563b9d"} Sep 19 08:20:53.440549 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.440042 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jhxfp" event={"ID":"7232a07f-d78f-4e05-b33d-9e28c67839e7","Type":"ContainerStarted","Data":"df51b072b46a767b9cdbdf30a62126d93ff54abe3956433a397e7e224edbfc8e"} Sep 19 08:20:53.441350 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.441258 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:53.441350 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.441325 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vzdf4" Sep 19 08:20:53.544228 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:53.544188 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 19 08:20:54.212833 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:54.212680 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-19T08:20:53.544208865Z","UUID":"b161bec9-c00e-42f4-8558-7d326ddfceb2","Handler":null,"Name":"","Endpoint":""} Sep 19 08:20:54.214856 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:54.214817 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 19 08:20:54.214856 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:54.214861 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 19 08:20:54.445809 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:54.445763 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" event={"ID":"8f8fee67-86f5-4706-a9f3-d368cece2657","Type":"ContainerStarted","Data":"d1b35ea363e6e53d98cc07da8e26f3aa247e1b3e11f442c893f89b11a33721af"} Sep 19 08:20:55.301340 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:55.301065 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:55.301340 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:55.301065 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:55.301587 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:55.301346 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:55.301587 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:55.301065 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:55.301587 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:55.301406 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:55.301587 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:55.301451 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:55.450809 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:55.450770 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" event={"ID":"8f8fee67-86f5-4706-a9f3-d368cece2657","Type":"ContainerStarted","Data":"f043fb6c2f5de750f6b80fa008c9ddf7ab4ae35b46213dc92eeb83e0ec491b2d"} Sep 19 08:20:55.454641 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:55.454611 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"20b015920fb48b84b4cb6173a47fbab44b2727d47b0384cb249e2575dd94e8df"} Sep 19 08:20:55.473403 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:55.473341 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h4n5k" podStartSLOduration=4.3279715979999995 podStartE2EDuration="24.473321438s" totalImagesPullingTime="20.14534984s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:55.47306261 +0000 UTC m=+24.684896468" watchObservedRunningTime="2026-09-19 08:20:55.473321438 +0000 UTC m=+24.685155296" Sep 19 08:20:55.473874 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:55.473835 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jhxfp" podStartSLOduration=6.738577602 podStartE2EDuration="24.473824212s" totalImagesPullingTime="17.73524661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:53.501584798 +0000 UTC m=+22.713418659" watchObservedRunningTime="2026-09-19 08:20:55.473824212 +0000 UTC m=+24.685658071" Sep 19 08:20:57.300928 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.300707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:57.300928 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.300711 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:57.300928 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.300714 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:57.301424 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:57.300989 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:57.301424 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:57.301032 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:57.301424 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:57.301106 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:57.460636 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.460602 2585 generic.go:356] "Generic (PLEG): container finished" podID="cacb029c-d5ba-4bdf-b8ae-cbd5b809de91" containerID="425e8cb0ed7e9565e9289722e7a6420c640bb2b54896220d06ca7bb3eed41b2b" exitCode=0 Sep 19 08:20:57.460854 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.460682 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" event={"ID":"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91","Type":"ContainerDied","Data":"425e8cb0ed7e9565e9289722e7a6420c640bb2b54896220d06ca7bb3eed41b2b"} Sep 19 08:20:57.467140 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.467116 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" event={"ID":"c0830ee0-b0ea-4e9d-bac7-3f61cc93712e","Type":"ContainerStarted","Data":"3e58840cfd5f52073a9c6a15fca4e7bdc1161d7432f808cc02f922357e90fcb3"} Sep 19 08:20:57.467493 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.467479 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:57.467732 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.467713 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:57.482342 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.482314 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:57.482698 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.482683 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:20:57.505750 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:57.505712 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" podStartSLOduration=8.762490131 podStartE2EDuration="26.505699943s" totalImagesPullingTime="17.743209812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:20:57.504339492 +0000 UTC m=+26.716173349" watchObservedRunningTime="2026-09-19 08:20:57.505699943 +0000 UTC m=+26.717533800" Sep 19 08:20:58.469375 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:58.469340 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 19 08:20:58.868654 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:58.868581 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l9799"] Sep 19 08:20:58.868773 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:58.868696 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:20:58.868812 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:58.868777 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:20:58.872658 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:58.872626 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7bd8s"] Sep 19 08:20:58.872761 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:58.872738 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:20:58.872855 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:58.872835 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:20:58.873326 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:58.873307 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mz86w"] Sep 19 08:20:58.873410 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:58.873393 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:20:58.873474 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:20:58.873459 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:20:59.473373 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:59.473336 2585 generic.go:356] "Generic (PLEG): container finished" podID="cacb029c-d5ba-4bdf-b8ae-cbd5b809de91" containerID="2828062b02bf2bb2376a55ed2b3f16947a83ae98b4192553da80bff25935e1f9" exitCode=0 Sep 19 08:20:59.473917 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:59.473419 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" event={"ID":"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91","Type":"ContainerDied","Data":"2828062b02bf2bb2376a55ed2b3f16947a83ae98b4192553da80bff25935e1f9"} Sep 19 08:20:59.473917 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:59.473516 2585 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 19 08:20:59.578355 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:20:59.578325 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:21:00.301069 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:00.300862 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:00.301275 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:00.300872 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:21:00.301275 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:00.301167 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:21:00.301275 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:00.300922 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:21:00.301275 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:00.301260 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:21:00.301460 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:00.301330 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:21:01.480936 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:01.480902 2585 generic.go:356] "Generic (PLEG): container finished" podID="cacb029c-d5ba-4bdf-b8ae-cbd5b809de91" containerID="af4f6aae677b2ec92aa4ccc47d14cd103edcdf5602ca2471e8c7448cd602f696" exitCode=0 Sep 19 08:21:01.481323 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:01.480962 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" event={"ID":"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91","Type":"ContainerDied","Data":"af4f6aae677b2ec92aa4ccc47d14cd103edcdf5602ca2471e8c7448cd602f696"} Sep 19 08:21:02.300479 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:02.300444 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:21:02.300680 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:02.300559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:02.300680 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:02.300564 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:21:02.300680 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:02.300590 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:21:02.300680 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:02.300670 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:21:02.300841 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:02.300735 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:21:04.301201 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:04.301156 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:04.301605 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:04.301156 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:21:04.301605 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:04.301273 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l9799" podUID="aa486a4a-823d-40ca-b1d1-df39a7a1baf8" Sep 19 08:21:04.301605 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:04.301156 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:21:04.301605 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:04.301428 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bd8s" podUID="30dd1f4f-1ad8-4cca-94f9-ae57d1e43449" Sep 19 08:21:04.301605 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:04.301335 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mz86w" podUID="91530dee-a144-497b-b63a-3b88b4878905" Sep 19 08:21:05.000350 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.000324 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:21:05.000432 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.000365 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:21:05.000496 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.000483 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:21:05.000560 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.000551 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs podName:30dd1f4f-1ad8-4cca-94f9-ae57d1e43449 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:37.00053601 +0000 UTC m=+66.212369856 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs") pod "network-metrics-daemon-7bd8s" (UID: "30dd1f4f-1ad8-4cca-94f9-ae57d1e43449") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 19 08:21:05.000608 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.000483 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 19 08:21:05.000637 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.000614 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret podName:91530dee-a144-497b-b63a-3b88b4878905 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:37.000601671 +0000 UTC m=+66.212435522 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret") pod "global-pull-secret-syncer-mz86w" (UID: "91530dee-a144-497b-b63a-3b88b4878905") : object "kube-system"/"original-pull-secret" not registered Sep 19 08:21:05.082303 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.082267 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-127.ec2.internal" event="NodeReady" Sep 19 08:21:05.082461 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.082432 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 19 08:21:05.101693 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.101662 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:05.101852 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.101831 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 19 08:21:05.101902 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.101858 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 19 08:21:05.101902 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.101869 2585 projected.go:196] Error preparing data for projected volume kube-api-access-kpbkq for pod openshift-network-diagnostics/network-check-target-l9799: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:21:05.101997 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.101921 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq podName:aa486a4a-823d-40ca-b1d1-df39a7a1baf8 nodeName:}" failed. No retries permitted until 2026-09-19 08:21:37.10190192 +0000 UTC m=+66.313735764 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kpbkq" (UniqueName: "kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq") pod "network-check-target-l9799" (UID: "aa486a4a-823d-40ca-b1d1-df39a7a1baf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 19 08:21:05.127305 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.127273 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-swsxs"] Sep 19 08:21:05.129739 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.129712 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.133375 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.133341 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 19 08:21:05.133697 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.133679 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cdf7v\"" Sep 19 08:21:05.133797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.133714 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 19 08:21:05.134804 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.134781 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v68l8"] Sep 19 08:21:05.136589 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.136437 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v68l8" Sep 19 08:21:05.141670 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.141650 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 19 08:21:05.141779 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.141659 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2fzkh\"" Sep 19 08:21:05.141843 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.141825 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 19 08:21:05.141940 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.141924 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 19 08:21:05.149056 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.149032 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-swsxs"] Sep 19 08:21:05.153046 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.153022 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v68l8"] Sep 19 08:21:05.303326 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.303244 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrh5\" (UniqueName: \"kubernetes.io/projected/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-kube-api-access-skrh5\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.303326 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.303287 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjszx\" (UniqueName: \"kubernetes.io/projected/9cd4ad62-dadf-42d7-8696-93041ca4dd12-kube-api-access-jjszx\") pod \"ingress-canary-v68l8\" (UID: \"9cd4ad62-dadf-42d7-8696-93041ca4dd12\") " pod="openshift-ingress-canary/ingress-canary-v68l8" Sep 19 08:21:05.303326 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.303323 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-config-volume\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.303982 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.303348 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.303982 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.303370 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-tmp-dir\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.303982 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.303439 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd4ad62-dadf-42d7-8696-93041ca4dd12-cert\") pod \"ingress-canary-v68l8\" (UID: \"9cd4ad62-dadf-42d7-8696-93041ca4dd12\") " pod="openshift-ingress-canary/ingress-canary-v68l8" Sep 19 08:21:05.404412 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.404375 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrh5\" (UniqueName: \"kubernetes.io/projected/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-kube-api-access-skrh5\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.404412 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.404419 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjszx\" (UniqueName: \"kubernetes.io/projected/9cd4ad62-dadf-42d7-8696-93041ca4dd12-kube-api-access-jjszx\") pod \"ingress-canary-v68l8\" (UID: \"9cd4ad62-dadf-42d7-8696-93041ca4dd12\") " pod="openshift-ingress-canary/ingress-canary-v68l8" Sep 19 08:21:05.404676 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.404466 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-config-volume\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.404676 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.404490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.404676 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.404510 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-tmp-dir\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.404676 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.404567 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd4ad62-dadf-42d7-8696-93041ca4dd12-cert\") pod \"ingress-canary-v68l8\" (UID: \"9cd4ad62-dadf-42d7-8696-93041ca4dd12\") " pod="openshift-ingress-canary/ingress-canary-v68l8" Sep 19 08:21:05.404676 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.404622 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 08:21:05.404911 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.404688 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls podName:d4fa1e4d-c5fe-4072-8e44-f40975acb77d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:05.904668676 +0000 UTC m=+35.116502525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls") pod "dns-default-swsxs" (UID: "d4fa1e4d-c5fe-4072-8e44-f40975acb77d") : secret "dns-default-metrics-tls" not found Sep 19 08:21:05.405009 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.404919 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-tmp-dir\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.405165 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.405144 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-config-volume\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.409280 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.409252 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd4ad62-dadf-42d7-8696-93041ca4dd12-cert\") pod \"ingress-canary-v68l8\" (UID: \"9cd4ad62-dadf-42d7-8696-93041ca4dd12\") " pod="openshift-ingress-canary/ingress-canary-v68l8" Sep 19 08:21:05.414196 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.414167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrh5\" (UniqueName: \"kubernetes.io/projected/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-kube-api-access-skrh5\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.414606 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.414586 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjszx\" (UniqueName: \"kubernetes.io/projected/9cd4ad62-dadf-42d7-8696-93041ca4dd12-kube-api-access-jjszx\") pod \"ingress-canary-v68l8\" (UID: \"9cd4ad62-dadf-42d7-8696-93041ca4dd12\") " pod="openshift-ingress-canary/ingress-canary-v68l8" Sep 19 08:21:05.448635 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.448602 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v68l8" Sep 19 08:21:05.907515 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:05.907478 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:05.907717 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.907622 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 08:21:05.907717 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:05.907688 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls podName:d4fa1e4d-c5fe-4072-8e44-f40975acb77d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:06.907671022 +0000 UTC m=+36.119504874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls") pod "dns-default-swsxs" (UID: "d4fa1e4d-c5fe-4072-8e44-f40975acb77d") : secret "dns-default-metrics-tls" not found Sep 19 08:21:06.300645 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.300559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:21:06.300795 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.300559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:21:06.300795 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.300559 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:06.304726 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.304703 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2vrmn\"" Sep 19 08:21:06.305155 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.304744 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 19 08:21:06.305155 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.304703 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 19 08:21:06.305155 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.304816 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 19 08:21:06.305155 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.304703 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-czqpg\"" Sep 19 08:21:06.305155 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.305080 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 19 08:21:06.800377 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.800209 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v68l8"] Sep 19 08:21:06.803898 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:06.803865 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd4ad62_dadf_42d7_8696_93041ca4dd12.slice/crio-e7761d5b160542dbf3ba0567eee3a9e18a4e4a2cdb9025f9233297450c6f006e WatchSource:0}: Error finding container e7761d5b160542dbf3ba0567eee3a9e18a4e4a2cdb9025f9233297450c6f006e: Status 404 returned error can't find the container with id e7761d5b160542dbf3ba0567eee3a9e18a4e4a2cdb9025f9233297450c6f006e Sep 19 08:21:06.913726 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.913693 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:06.913846 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:06.913834 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 08:21:06.913902 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:06.913892 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls podName:d4fa1e4d-c5fe-4072-8e44-f40975acb77d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.913877661 +0000 UTC m=+38.125711501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls") pod "dns-default-swsxs" (UID: "d4fa1e4d-c5fe-4072-8e44-f40975acb77d") : secret "dns-default-metrics-tls" not found Sep 19 08:21:06.957610 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.957577 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-268wh"] Sep 19 08:21:06.959534 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.959500 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:06.962485 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.962463 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 19 08:21:06.962619 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.962470 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 19 08:21:06.962619 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.962472 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 19 08:21:06.962843 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.962828 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 19 08:21:06.963880 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.963860 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8tfpd\"" Sep 19 08:21:06.968947 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.968929 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 19 08:21:06.970757 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:06.970737 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-268wh"] Sep 19 08:21:07.054042 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.054011 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-56868b9754-pt5rr"] Sep 19 08:21:07.055720 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.055705 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.058489 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.058472 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 19 08:21:07.058679 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.058667 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 19 08:21:07.058774 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.058759 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 19 08:21:07.058835 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.058770 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wc64m\"" Sep 19 08:21:07.058835 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.058767 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 19 08:21:07.058922 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.058899 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 19 08:21:07.059100 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.059084 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 19 08:21:07.071367 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.071340 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56868b9754-pt5rr"] Sep 19 08:21:07.116222 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.116132 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-snapshots\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.116222 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.116176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26xvx\" (UniqueName: \"kubernetes.io/projected/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-kube-api-access-26xvx\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.116407 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.116263 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-tmp\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.116407 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.116300 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.116407 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.116367 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-serving-cert\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.116407 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.116386 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.217889 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.217847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218053 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.217916 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj79q\" (UniqueName: \"kubernetes.io/projected/37d3bf81-c5c5-4807-8418-ea8dc81793ca-kube-api-access-dj79q\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.218053 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.217948 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-serving-cert\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218053 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.217974 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218053 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.217996 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-stats-auth\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.218288 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218061 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.218288 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218115 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-snapshots\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218288 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218160 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26xvx\" (UniqueName: \"kubernetes.io/projected/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-kube-api-access-26xvx\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218320 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-default-certificate\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.218442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.218442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218413 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-tmp\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218566 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-service-ca-bundle\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218779 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218752 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-tmp\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218847 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218798 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-snapshots\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.218898 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.218867 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-trusted-ca-bundle\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.221464 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.221441 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-serving-cert\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.227838 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.227813 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26xvx\" (UniqueName: \"kubernetes.io/projected/b1e03626-0f6b-41e4-a0c3-9e19c2d2de72-kube-api-access-26xvx\") pod \"insights-operator-6f9c644dd9-268wh\" (UID: \"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72\") " pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.268901 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.268864 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-6f9c644dd9-268wh" Sep 19 08:21:07.319799 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.319726 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj79q\" (UniqueName: \"kubernetes.io/projected/37d3bf81-c5c5-4807-8418-ea8dc81793ca-kube-api-access-dj79q\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.319799 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.319779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-stats-auth\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.320633 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.319809 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.320633 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.319846 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-default-certificate\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.320633 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.319870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.320633 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:07.320060 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:07.820039166 +0000 UTC m=+37.031873023 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : configmap references non-existent config key: service-ca.crt Sep 19 08:21:07.321646 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:07.321424 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 08:21:07.321646 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:07.321493 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:07.821474776 +0000 UTC m=+37.033308615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : secret "router-metrics-certs-default" not found Sep 19 08:21:07.324740 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.324692 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-default-certificate\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.326824 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.326781 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-stats-auth\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.332886 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.332828 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj79q\" (UniqueName: \"kubernetes.io/projected/37d3bf81-c5c5-4807-8418-ea8dc81793ca-kube-api-access-dj79q\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.496000 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.495959 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v68l8" event={"ID":"9cd4ad62-dadf-42d7-8696-93041ca4dd12","Type":"ContainerStarted","Data":"e7761d5b160542dbf3ba0567eee3a9e18a4e4a2cdb9025f9233297450c6f006e"} Sep 19 08:21:07.500247 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.499996 2585 generic.go:356] "Generic (PLEG): container finished" podID="cacb029c-d5ba-4bdf-b8ae-cbd5b809de91" containerID="e3188c525bd10c665dede994ee7786ac6639f12926be75aa8e7c2957e1845eab" exitCode=0 Sep 19 08:21:07.500247 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.500049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" event={"ID":"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91","Type":"ContainerDied","Data":"e3188c525bd10c665dede994ee7786ac6639f12926be75aa8e7c2957e1845eab"} Sep 19 08:21:07.543373 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.542804 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-6f9c644dd9-268wh"] Sep 19 08:21:07.548263 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:07.548233 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e03626_0f6b_41e4_a0c3_9e19c2d2de72.slice/crio-93b2797b8604bbfc1b621d0ee7934d17ce34f8c243b84f6900587ba4be730cb0 WatchSource:0}: Error finding container 93b2797b8604bbfc1b621d0ee7934d17ce34f8c243b84f6900587ba4be730cb0: Status 404 returned error can't find the container with id 93b2797b8604bbfc1b621d0ee7934d17ce34f8c243b84f6900587ba4be730cb0 Sep 19 08:21:07.823465 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.823382 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.823634 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:07.823508 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:07.823634 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:07.823592 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.823569744 +0000 UTC m=+38.035403593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : configmap references non-existent config key: service-ca.crt Sep 19 08:21:07.823773 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:07.823650 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 08:21:07.823773 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:07.823713 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:08.823695754 +0000 UTC m=+38.035529604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : secret "router-metrics-certs-default" not found Sep 19 08:21:08.504359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:08.504320 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-268wh" event={"ID":"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72","Type":"ContainerStarted","Data":"93b2797b8604bbfc1b621d0ee7934d17ce34f8c243b84f6900587ba4be730cb0"} Sep 19 08:21:08.508264 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:08.508236 2585 generic.go:356] "Generic (PLEG): container finished" podID="cacb029c-d5ba-4bdf-b8ae-cbd5b809de91" containerID="729936d6be7853d4d8d2bdbea0b41d469f05eb4864acb416c96a740fd14cc91c" exitCode=0 Sep 19 08:21:08.508389 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:08.508296 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" event={"ID":"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91","Type":"ContainerDied","Data":"729936d6be7853d4d8d2bdbea0b41d469f05eb4864acb416c96a740fd14cc91c"} Sep 19 08:21:08.831138 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:08.830848 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:08.831138 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:08.831129 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:08.831363 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:08.831032 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 08:21:08.831363 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:08.831235 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:10.831212373 +0000 UTC m=+40.043046213 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : secret "router-metrics-certs-default" not found Sep 19 08:21:08.831363 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:08.831284 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:10.831269796 +0000 UTC m=+40.043103631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : configmap references non-existent config key: service-ca.crt Sep 19 08:21:08.932306 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:08.932264 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:08.932481 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:08.932440 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 08:21:08.932552 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:08.932533 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls podName:d4fa1e4d-c5fe-4072-8e44-f40975acb77d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:12.932499252 +0000 UTC m=+42.144333090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls") pod "dns-default-swsxs" (UID: "d4fa1e4d-c5fe-4072-8e44-f40975acb77d") : secret "dns-default-metrics-tls" not found Sep 19 08:21:09.515060 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:09.514941 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" event={"ID":"cacb029c-d5ba-4bdf-b8ae-cbd5b809de91","Type":"ContainerStarted","Data":"7b5bf8fcc6f28418db4c037c973bd0a9fe6e0b0ab59ae52410d6771a3df57cf8"} Sep 19 08:21:09.540299 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:09.540246 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-rwwtw" podStartSLOduration=8.951448958 podStartE2EDuration="38.540231287s" totalImagesPullingTime="29.588782329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:09.538578493 +0000 UTC m=+38.750412402" watchObservedRunningTime="2026-09-19 08:21:09.540231287 +0000 UTC m=+38.752065143" Sep 19 08:21:10.518683 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:10.518645 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v68l8" event={"ID":"9cd4ad62-dadf-42d7-8696-93041ca4dd12","Type":"ContainerStarted","Data":"c9dd173a11a6a902d4c9cc64b7a9978388d3d5c88b6a5d5910b7d084cb1eee9a"} Sep 19 08:21:10.520326 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:10.520292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-268wh" event={"ID":"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72","Type":"ContainerStarted","Data":"905299bf12fe34ef198e74a5fb1022ff61dab2c9ff911d8be93ff0d08f336c87"} Sep 19 08:21:10.535857 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:10.535813 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v68l8" podStartSLOduration=2.360804651 podStartE2EDuration="5.535793213s" totalImagesPullingTime="3.174988562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:10.534190925 +0000 UTC m=+39.746024783" watchObservedRunningTime="2026-09-19 08:21:10.535793213 +0000 UTC m=+39.747627070" Sep 19 08:21:10.552005 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:10.551958 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-6f9c644dd9-268wh" podStartSLOduration=1.994952628 podStartE2EDuration="4.551942439s" totalImagesPullingTime="2.556989811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:10.55127096 +0000 UTC m=+39.763104818" watchObservedRunningTime="2026-09-19 08:21:10.551942439 +0000 UTC m=+39.763776296" Sep 19 08:21:10.848024 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:10.847936 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:10.848024 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:10.847988 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:10.848202 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:10.848081 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 08:21:10.848202 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:10.848151 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:14.848135149 +0000 UTC m=+44.059969004 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : configmap references non-existent config key: service-ca.crt Sep 19 08:21:10.848202 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:10.848168 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:14.848161491 +0000 UTC m=+44.059995331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : secret "router-metrics-certs-default" not found Sep 19 08:21:11.658302 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.658275 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fkbpb"] Sep 19 08:21:11.679540 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.679502 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fkbpb"] Sep 19 08:21:11.679664 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.679647 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.682218 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.682190 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 19 08:21:11.682496 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.682475 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 19 08:21:11.682631 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.682566 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jw55w\"" Sep 19 08:21:11.856138 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.856093 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms4pr\" (UniqueName: \"kubernetes.io/projected/f32eb384-ceda-409c-be43-f0f8c014541a-kube-api-access-ms4pr\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.856138 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.856135 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f32eb384-ceda-409c-be43-f0f8c014541a-crio-socket\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.856337 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.856163 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.856337 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.856300 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f32eb384-ceda-409c-be43-f0f8c014541a-data-volume\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.856410 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.856348 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.957293 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.957207 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f32eb384-ceda-409c-be43-f0f8c014541a-data-volume\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.957293 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.957241 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.957463 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:11.957341 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 08:21:11.957463 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:11.957394 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:12.45737894 +0000 UTC m=+41.669212775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : configmap "kube-rbac-proxy" not found Sep 19 08:21:11.957463 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.957392 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms4pr\" (UniqueName: \"kubernetes.io/projected/f32eb384-ceda-409c-be43-f0f8c014541a-kube-api-access-ms4pr\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.957463 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.957446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f32eb384-ceda-409c-be43-f0f8c014541a-crio-socket\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.957664 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.957474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.957664 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.957627 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f32eb384-ceda-409c-be43-f0f8c014541a-data-volume\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.957664 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:11.957646 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 08:21:11.957841 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.957663 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f32eb384-ceda-409c-be43-f0f8c014541a-crio-socket\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:11.957841 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:11.957721 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:12.457700536 +0000 UTC m=+41.669534373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : secret "insights-runtime-extractor-tls" not found Sep 19 08:21:11.969612 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:11.969582 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms4pr\" (UniqueName: \"kubernetes.io/projected/f32eb384-ceda-409c-be43-f0f8c014541a-kube-api-access-ms4pr\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:12.462674 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:12.462626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:12.462865 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:12.462754 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 08:21:12.462865 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:12.462759 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:12.462865 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:12.462822 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:13.462804608 +0000 UTC m=+42.674638465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : configmap "kube-rbac-proxy" not found Sep 19 08:21:12.462976 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:12.462904 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 08:21:12.462976 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:12.462966 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:13.462949966 +0000 UTC m=+42.674783801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : secret "insights-runtime-extractor-tls" not found Sep 19 08:21:12.966407 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:12.966373 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:12.966797 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:12.966479 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 08:21:12.966797 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:12.966545 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls podName:d4fa1e4d-c5fe-4072-8e44-f40975acb77d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:20.966515874 +0000 UTC m=+50.178349710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls") pod "dns-default-swsxs" (UID: "d4fa1e4d-c5fe-4072-8e44-f40975acb77d") : secret "dns-default-metrics-tls" not found Sep 19 08:21:13.188693 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:13.188665 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9c7kk_ff2f92e7-7821-4a5c-9807-e81088d7e25f/dns-node-resolver/0.log" Sep 19 08:21:13.470138 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:13.470097 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:13.470316 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:13.470189 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:13.470316 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:13.470255 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 08:21:13.470316 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:13.470275 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 08:21:13.470410 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:13.470325 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:15.470312304 +0000 UTC m=+44.682146141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : configmap "kube-rbac-proxy" not found Sep 19 08:21:13.470410 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:13.470339 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:15.470333564 +0000 UTC m=+44.682167399 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : secret "insights-runtime-extractor-tls" not found Sep 19 08:21:13.989848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:13.989821 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvk4d_492622cf-75d6-416d-bc84-1ba3effd3382/node-ca/0.log" Sep 19 08:21:14.789536 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:14.789495 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v68l8_9cd4ad62-dadf-42d7-8696-93041ca4dd12/serve-healthcheck-canary/0.log" Sep 19 08:21:14.881889 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:14.881850 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:14.882064 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:14.881905 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:14.882064 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:14.882006 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 08:21:14.882064 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:14.882056 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:22.882043884 +0000 UTC m=+52.093877719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : configmap references non-existent config key: service-ca.crt Sep 19 08:21:14.882191 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:14.882071 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:22.882064974 +0000 UTC m=+52.093898809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : secret "router-metrics-certs-default" not found Sep 19 08:21:15.486439 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:15.486408 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:15.486821 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:15.486549 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 08:21:15.486821 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:15.486610 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:19.486595346 +0000 UTC m=+48.698429180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : configmap "kube-rbac-proxy" not found Sep 19 08:21:15.486821 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:15.486623 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 08:21:15.486821 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:15.486663 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:19.486652599 +0000 UTC m=+48.698486434 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : secret "insights-runtime-extractor-tls" not found Sep 19 08:21:15.486821 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:15.486559 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:16.977587 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:16.977556 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc"] Sep 19 08:21:17.011912 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.011886 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc"] Sep 19 08:21:17.012024 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.011973 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc" Sep 19 08:21:17.015095 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.015073 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 19 08:21:17.015325 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.015310 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 19 08:21:17.016277 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.016256 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6qg2r\"" Sep 19 08:21:17.089881 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.089848 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-nvgxc"] Sep 19 08:21:17.097857 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.097825 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mkr5\" (UniqueName: \"kubernetes.io/projected/f34f604c-0f98-4be4-91aa-8ec50ac41ecc-kube-api-access-6mkr5\") pod \"volume-data-source-validator-5c4db8bdf9-8tzfc\" (UID: \"f34f604c-0f98-4be4-91aa-8ec50ac41ecc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc" Sep 19 08:21:17.113629 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.113605 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-nvgxc"] Sep 19 08:21:17.113754 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.113691 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.116493 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.116473 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 19 08:21:17.116643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.116516 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 19 08:21:17.117066 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.116984 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-tsv2f\"" Sep 19 08:21:17.117333 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.117005 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 19 08:21:17.118145 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.118121 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 19 08:21:17.134170 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.134149 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 19 08:21:17.187465 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.187441 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2"] Sep 19 08:21:17.198442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.198415 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b88c6ebb-8599-443a-a656-8aa7f8079fd3-serving-cert\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.198546 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.198481 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mkr5\" (UniqueName: \"kubernetes.io/projected/f34f604c-0f98-4be4-91aa-8ec50ac41ecc-kube-api-access-6mkr5\") pod \"volume-data-source-validator-5c4db8bdf9-8tzfc\" (UID: \"f34f604c-0f98-4be4-91aa-8ec50ac41ecc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc" Sep 19 08:21:17.198605 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.198544 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87768\" (UniqueName: \"kubernetes.io/projected/b88c6ebb-8599-443a-a656-8aa7f8079fd3-kube-api-access-87768\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.198643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.198636 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b88c6ebb-8599-443a-a656-8aa7f8079fd3-config\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.198681 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.198667 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b88c6ebb-8599-443a-a656-8aa7f8079fd3-trusted-ca\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.203577 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.203558 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2"] Sep 19 08:21:17.203675 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.203636 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.206269 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.206250 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 19 08:21:17.206367 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.206288 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 19 08:21:17.206367 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.206302 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 19 08:21:17.206497 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.206368 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 19 08:21:17.206697 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.206681 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-fw2h4\"" Sep 19 08:21:17.217777 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.217761 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mkr5\" (UniqueName: \"kubernetes.io/projected/f34f604c-0f98-4be4-91aa-8ec50ac41ecc-kube-api-access-6mkr5\") pod \"volume-data-source-validator-5c4db8bdf9-8tzfc\" (UID: \"f34f604c-0f98-4be4-91aa-8ec50ac41ecc\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc" Sep 19 08:21:17.299650 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.299598 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-967bv\" (UniqueName: \"kubernetes.io/projected/10f7eb02-068f-444d-8695-17e0d5f7609e-kube-api-access-967bv\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.299650 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.299640 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b88c6ebb-8599-443a-a656-8aa7f8079fd3-serving-cert\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.299800 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.299676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10f7eb02-068f-444d-8695-17e0d5f7609e-serving-cert\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.299800 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.299703 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10f7eb02-068f-444d-8695-17e0d5f7609e-config\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.299800 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.299736 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b88c6ebb-8599-443a-a656-8aa7f8079fd3-trusted-ca\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.299800 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.299789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87768\" (UniqueName: \"kubernetes.io/projected/b88c6ebb-8599-443a-a656-8aa7f8079fd3-kube-api-access-87768\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.299954 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.299835 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b88c6ebb-8599-443a-a656-8aa7f8079fd3-config\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.300492 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.300475 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b88c6ebb-8599-443a-a656-8aa7f8079fd3-config\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.300611 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.300595 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b88c6ebb-8599-443a-a656-8aa7f8079fd3-trusted-ca\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.301804 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.301775 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b88c6ebb-8599-443a-a656-8aa7f8079fd3-serving-cert\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.308599 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.308580 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87768\" (UniqueName: \"kubernetes.io/projected/b88c6ebb-8599-443a-a656-8aa7f8079fd3-kube-api-access-87768\") pod \"console-operator-fd4bb46c-nvgxc\" (UID: \"b88c6ebb-8599-443a-a656-8aa7f8079fd3\") " pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.320410 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.320390 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc" Sep 19 08:21:17.400281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.400208 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-967bv\" (UniqueName: \"kubernetes.io/projected/10f7eb02-068f-444d-8695-17e0d5f7609e-kube-api-access-967bv\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.400406 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.400326 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10f7eb02-068f-444d-8695-17e0d5f7609e-serving-cert\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.400406 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.400375 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10f7eb02-068f-444d-8695-17e0d5f7609e-config\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.401252 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.401227 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10f7eb02-068f-444d-8695-17e0d5f7609e-config\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.402837 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.402819 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10f7eb02-068f-444d-8695-17e0d5f7609e-serving-cert\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.408367 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.408349 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-967bv\" (UniqueName: \"kubernetes.io/projected/10f7eb02-068f-444d-8695-17e0d5f7609e-kube-api-access-967bv\") pod \"service-ca-operator-57cdc4b768-2tss2\" (UID: \"10f7eb02-068f-444d-8695-17e0d5f7609e\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.423530 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.423497 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:17.516810 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.516777 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" Sep 19 08:21:17.618324 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.618300 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc"] Sep 19 08:21:17.621119 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:17.621095 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf34f604c_0f98_4be4_91aa_8ec50ac41ecc.slice/crio-fa7c423ab9996972767a5473c6f071e0a9e3b3feac63f5a297400257d44c5a65 WatchSource:0}: Error finding container fa7c423ab9996972767a5473c6f071e0a9e3b3feac63f5a297400257d44c5a65: Status 404 returned error can't find the container with id fa7c423ab9996972767a5473c6f071e0a9e3b3feac63f5a297400257d44c5a65 Sep 19 08:21:17.722262 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.722236 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-fd4bb46c-nvgxc"] Sep 19 08:21:17.725960 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:17.725935 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88c6ebb_8599_443a_a656_8aa7f8079fd3.slice/crio-043fe899dc9a85d235815dcc5d02fb1b40d63b13357aab1c92c2f19a41166ab8 WatchSource:0}: Error finding container 043fe899dc9a85d235815dcc5d02fb1b40d63b13357aab1c92c2f19a41166ab8: Status 404 returned error can't find the container with id 043fe899dc9a85d235815dcc5d02fb1b40d63b13357aab1c92c2f19a41166ab8 Sep 19 08:21:17.816204 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:17.816138 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2"] Sep 19 08:21:17.818943 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:17.818918 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10f7eb02_068f_444d_8695_17e0d5f7609e.slice/crio-973a01cfd2c87dee4adb6cfaa0135fe1d25d837a10767c4f77fe2b66a7593c48 WatchSource:0}: Error finding container 973a01cfd2c87dee4adb6cfaa0135fe1d25d837a10767c4f77fe2b66a7593c48: Status 404 returned error can't find the container with id 973a01cfd2c87dee4adb6cfaa0135fe1d25d837a10767c4f77fe2b66a7593c48 Sep 19 08:21:18.540852 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:18.540814 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc" event={"ID":"f34f604c-0f98-4be4-91aa-8ec50ac41ecc","Type":"ContainerStarted","Data":"fa7c423ab9996972767a5473c6f071e0a9e3b3feac63f5a297400257d44c5a65"} Sep 19 08:21:18.543026 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:18.542973 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" event={"ID":"10f7eb02-068f-444d-8695-17e0d5f7609e","Type":"ContainerStarted","Data":"973a01cfd2c87dee4adb6cfaa0135fe1d25d837a10767c4f77fe2b66a7593c48"} Sep 19 08:21:18.544418 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:18.544394 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" event={"ID":"b88c6ebb-8599-443a-a656-8aa7f8079fd3","Type":"ContainerStarted","Data":"043fe899dc9a85d235815dcc5d02fb1b40d63b13357aab1c92c2f19a41166ab8"} Sep 19 08:21:19.513768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:19.513724 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:19.513957 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:19.513896 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:19.513957 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:19.513902 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 08:21:19.514065 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:19.513978 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:27.513957495 +0000 UTC m=+56.725791342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : secret "insights-runtime-extractor-tls" not found Sep 19 08:21:19.514065 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:19.514019 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 19 08:21:19.514169 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:19.514080 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:27.514059369 +0000 UTC m=+56.725893226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : configmap "kube-rbac-proxy" not found Sep 19 08:21:20.933687 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:20.933658 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm"] Sep 19 08:21:20.946973 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:20.946945 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" Sep 19 08:21:20.949308 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:20.949283 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm"] Sep 19 08:21:20.949769 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:20.949750 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 19 08:21:20.949866 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:20.949750 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 19 08:21:20.950857 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:20.950842 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tv52j\"" Sep 19 08:21:21.026234 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.026203 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:21.026358 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.026257 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcmrx\" (UniqueName: \"kubernetes.io/projected/84b85994-b1ef-49b3-bc7c-dca7867d330b-kube-api-access-fcmrx\") pod \"migrator-76794cbd6c-srhjm\" (UID: \"84b85994-b1ef-49b3-bc7c-dca7867d330b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" Sep 19 08:21:21.026358 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:21.026337 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 19 08:21:21.026450 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:21.026414 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls podName:d4fa1e4d-c5fe-4072-8e44-f40975acb77d nodeName:}" failed. No retries permitted until 2026-09-19 08:21:37.026391216 +0000 UTC m=+66.238225064 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls") pod "dns-default-swsxs" (UID: "d4fa1e4d-c5fe-4072-8e44-f40975acb77d") : secret "dns-default-metrics-tls" not found Sep 19 08:21:21.127498 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.127463 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcmrx\" (UniqueName: \"kubernetes.io/projected/84b85994-b1ef-49b3-bc7c-dca7867d330b-kube-api-access-fcmrx\") pod \"migrator-76794cbd6c-srhjm\" (UID: \"84b85994-b1ef-49b3-bc7c-dca7867d330b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" Sep 19 08:21:21.136172 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.136150 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcmrx\" (UniqueName: \"kubernetes.io/projected/84b85994-b1ef-49b3-bc7c-dca7867d330b-kube-api-access-fcmrx\") pod \"migrator-76794cbd6c-srhjm\" (UID: \"84b85994-b1ef-49b3-bc7c-dca7867d330b\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" Sep 19 08:21:21.257062 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.256983 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" Sep 19 08:21:21.546083 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.546008 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm"] Sep 19 08:21:21.549085 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:21.549060 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b85994_b1ef_49b3_bc7c_dca7867d330b.slice/crio-e408c8c192b6fd54820e9e4d049732e7f2f8c997fe21af670caccf65b6c83eb2 WatchSource:0}: Error finding container e408c8c192b6fd54820e9e4d049732e7f2f8c997fe21af670caccf65b6c83eb2: Status 404 returned error can't find the container with id e408c8c192b6fd54820e9e4d049732e7f2f8c997fe21af670caccf65b6c83eb2 Sep 19 08:21:21.553406 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.553378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc" event={"ID":"f34f604c-0f98-4be4-91aa-8ec50ac41ecc","Type":"ContainerStarted","Data":"a04b9dec5a22e8df97ebba7852d5ec216b0a7070e9f2a1580dc5522d36baedbe"} Sep 19 08:21:21.554751 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.554693 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" event={"ID":"10f7eb02-068f-444d-8695-17e0d5f7609e","Type":"ContainerStarted","Data":"a551002591750e260bc9f996c6dddfe2e3c6524dc3fa1b53ae5f0b1475b397cd"} Sep 19 08:21:21.556348 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.556330 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/0.log" Sep 19 08:21:21.556422 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.556363 2585 generic.go:356] "Generic (PLEG): container finished" podID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" containerID="ace2b86d3a9335521bd304f5e712964119f7ad33d2e394ed23f2efe4a0dc0884" exitCode=255 Sep 19 08:21:21.556471 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.556443 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" event={"ID":"b88c6ebb-8599-443a-a656-8aa7f8079fd3","Type":"ContainerDied","Data":"ace2b86d3a9335521bd304f5e712964119f7ad33d2e394ed23f2efe4a0dc0884"} Sep 19 08:21:21.556689 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.556674 2585 scope.go:117] "RemoveContainer" containerID="ace2b86d3a9335521bd304f5e712964119f7ad33d2e394ed23f2efe4a0dc0884" Sep 19 08:21:21.557699 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.557671 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" event={"ID":"84b85994-b1ef-49b3-bc7c-dca7867d330b","Type":"ContainerStarted","Data":"e408c8c192b6fd54820e9e4d049732e7f2f8c997fe21af670caccf65b6c83eb2"} Sep 19 08:21:21.587276 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:21.587235 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5c4db8bdf9-8tzfc" podStartSLOduration=2.67842488 podStartE2EDuration="5.587220989s" totalImagesPullingTime="2.908796109s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:21.568274143 +0000 UTC m=+50.780108002" watchObservedRunningTime="2026-09-19 08:21:21.587220989 +0000 UTC m=+50.799054855" Sep 19 08:21:22.563566 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.563513 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/1.log" Sep 19 08:21:22.564019 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.563981 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/0.log" Sep 19 08:21:22.564081 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.564014 2585 generic.go:356] "Generic (PLEG): container finished" podID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" containerID="f6900d30840f9273a2222aaae1b04f5d0c84ce5b1e8c768568cfc99891a1aec9" exitCode=255 Sep 19 08:21:22.564136 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.564098 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" event={"ID":"b88c6ebb-8599-443a-a656-8aa7f8079fd3","Type":"ContainerDied","Data":"f6900d30840f9273a2222aaae1b04f5d0c84ce5b1e8c768568cfc99891a1aec9"} Sep 19 08:21:22.564191 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.564183 2585 scope.go:117] "RemoveContainer" containerID="ace2b86d3a9335521bd304f5e712964119f7ad33d2e394ed23f2efe4a0dc0884" Sep 19 08:21:22.564395 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.564376 2585 scope.go:117] "RemoveContainer" containerID="f6900d30840f9273a2222aaae1b04f5d0c84ce5b1e8c768568cfc99891a1aec9" Sep 19 08:21:22.564634 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:22.564601 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-nvgxc_openshift-console-operator(b88c6ebb-8599-443a-a656-8aa7f8079fd3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" podUID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" Sep 19 08:21:22.580339 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.580288 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" podStartSLOduration=2.839338414 podStartE2EDuration="5.580271982s" totalImagesPullingTime="2.740933568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:21.607970998 +0000 UTC m=+50.819804854" watchObservedRunningTime="2026-09-19 08:21:22.580271982 +0000 UTC m=+51.792105840" Sep 19 08:21:22.942362 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.942328 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:22.942545 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:22.942490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:22.942599 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:22.942552 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:38.942507689 +0000 UTC m=+68.154341550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : configmap references non-existent config key: service-ca.crt Sep 19 08:21:22.942654 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:22.942603 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 19 08:21:22.942688 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:22.942654 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs podName:37d3bf81-c5c5-4807-8418-ea8dc81793ca nodeName:}" failed. No retries permitted until 2026-09-19 08:21:38.942638364 +0000 UTC m=+68.154472207 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs") pod "router-default-56868b9754-pt5rr" (UID: "37d3bf81-c5c5-4807-8418-ea8dc81793ca") : secret "router-metrics-certs-default" not found Sep 19 08:21:23.569701 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:23.569618 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/1.log" Sep 19 08:21:23.570105 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:23.569989 2585 scope.go:117] "RemoveContainer" containerID="f6900d30840f9273a2222aaae1b04f5d0c84ce5b1e8c768568cfc99891a1aec9" Sep 19 08:21:23.570187 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:23.570164 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-nvgxc_openshift-console-operator(b88c6ebb-8599-443a-a656-8aa7f8079fd3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" podUID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" Sep 19 08:21:23.571512 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:23.571487 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" event={"ID":"84b85994-b1ef-49b3-bc7c-dca7867d330b","Type":"ContainerStarted","Data":"2bf8511c74d2a884cb7ecacd953e8746682f983b13391ec32e86758ea08b513f"} Sep 19 08:21:23.571649 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:23.571556 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" event={"ID":"84b85994-b1ef-49b3-bc7c-dca7867d330b","Type":"ContainerStarted","Data":"12e20bbece852ff6b830f6ac89f4e0ef5a502e8cb049076c225219b7a767368e"} Sep 19 08:21:23.627318 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:23.627268 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-srhjm" podStartSLOduration=2.065600779 podStartE2EDuration="3.627252957s" totalImagesPullingTime="1.561652178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:23.62576174 +0000 UTC m=+52.837595607" watchObservedRunningTime="2026-09-19 08:21:23.627252957 +0000 UTC m=+52.839086813" Sep 19 08:21:27.424018 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:27.423979 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:27.424437 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:27.424042 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:27.424437 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:27.424351 2585 scope.go:117] "RemoveContainer" containerID="f6900d30840f9273a2222aaae1b04f5d0c84ce5b1e8c768568cfc99891a1aec9" Sep 19 08:21:27.424506 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:27.424485 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-fd4bb46c-nvgxc_openshift-console-operator(b88c6ebb-8599-443a-a656-8aa7f8079fd3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" podUID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" Sep 19 08:21:27.582298 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:27.582258 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:27.582514 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:27.582345 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:27.582514 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:27.582487 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 19 08:21:27.582668 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:27.582561 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls podName:f32eb384-ceda-409c-be43-f0f8c014541a nodeName:}" failed. No retries permitted until 2026-09-19 08:21:43.582542948 +0000 UTC m=+72.794376788 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fkbpb" (UID: "f32eb384-ceda-409c-be43-f0f8c014541a") : secret "insights-runtime-extractor-tls" not found Sep 19 08:21:27.582903 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:27.582878 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f32eb384-ceda-409c-be43-f0f8c014541a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:30.493779 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:30.493751 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-czp6j" Sep 19 08:21:37.055783 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.055688 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:21:37.055783 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.055777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:21:37.056279 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.055807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:37.058016 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.057997 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d4fa1e4d-c5fe-4072-8e44-f40975acb77d-metrics-tls\") pod \"dns-default-swsxs\" (UID: \"d4fa1e4d-c5fe-4072-8e44-f40975acb77d\") " pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:37.058649 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.058629 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 19 08:21:37.059681 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.059664 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 19 08:21:37.068981 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.068956 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30dd1f4f-1ad8-4cca-94f9-ae57d1e43449-metrics-certs\") pod \"network-metrics-daemon-7bd8s\" (UID: \"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449\") " pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:21:37.069086 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.068964 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/91530dee-a144-497b-b63a-3b88b4878905-original-pull-secret\") pod \"global-pull-secret-syncer-mz86w\" (UID: \"91530dee-a144-497b-b63a-3b88b4878905\") " pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:21:37.156758 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.156722 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:37.159917 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.159898 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 19 08:21:37.169759 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.169740 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 19 08:21:37.180193 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.180171 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbkq\" (UniqueName: \"kubernetes.io/projected/aa486a4a-823d-40ca-b1d1-df39a7a1baf8-kube-api-access-kpbkq\") pod \"network-check-target-l9799\" (UID: \"aa486a4a-823d-40ca-b1d1-df39a7a1baf8\") " pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:37.213931 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.213901 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2vrmn\"" Sep 19 08:21:37.218157 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.218142 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mz86w" Sep 19 08:21:37.221259 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.221243 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bd8s" Sep 19 08:21:37.226015 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.226001 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-czqpg\"" Sep 19 08:21:37.234212 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.234192 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:37.243100 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.243079 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cdf7v\"" Sep 19 08:21:37.251215 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.251193 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:37.511014 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.510990 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mz86w"] Sep 19 08:21:37.513337 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:37.513307 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91530dee_a144_497b_b63a_3b88b4878905.slice/crio-458acb3c56b97c48ca9ba842609c3f122a76aeb67f8b53f3e50527beb870e72c WatchSource:0}: Error finding container 458acb3c56b97c48ca9ba842609c3f122a76aeb67f8b53f3e50527beb870e72c: Status 404 returned error can't find the container with id 458acb3c56b97c48ca9ba842609c3f122a76aeb67f8b53f3e50527beb870e72c Sep 19 08:21:37.539792 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.539769 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7bd8s"] Sep 19 08:21:37.541956 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:37.541933 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30dd1f4f_1ad8_4cca_94f9_ae57d1e43449.slice/crio-33db245ae45d4d6771caf89cfc0814b6574a772e84b2763a6be5e0c8b0f7e3da WatchSource:0}: Error finding container 33db245ae45d4d6771caf89cfc0814b6574a772e84b2763a6be5e0c8b0f7e3da: Status 404 returned error can't find the container with id 33db245ae45d4d6771caf89cfc0814b6574a772e84b2763a6be5e0c8b0f7e3da Sep 19 08:21:37.569099 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.569039 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-swsxs"] Sep 19 08:21:37.571830 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:37.571807 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fa1e4d_c5fe_4072_8e44_f40975acb77d.slice/crio-16d09243e8ae725af0561f3bc0a18899a877d1acb367151c9821b5820053ee07 WatchSource:0}: Error finding container 16d09243e8ae725af0561f3bc0a18899a877d1acb367151c9821b5820053ee07: Status 404 returned error can't find the container with id 16d09243e8ae725af0561f3bc0a18899a877d1acb367151c9821b5820053ee07 Sep 19 08:21:37.601579 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.601553 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l9799"] Sep 19 08:21:37.604219 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:37.604198 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa486a4a_823d_40ca_b1d1_df39a7a1baf8.slice/crio-0960197eab84c8e99160b2eeb3538f6d396dce672268dc5784938c961c3caa4e WatchSource:0}: Error finding container 0960197eab84c8e99160b2eeb3538f6d396dce672268dc5784938c961c3caa4e: Status 404 returned error can't find the container with id 0960197eab84c8e99160b2eeb3538f6d396dce672268dc5784938c961c3caa4e Sep 19 08:21:37.617107 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.617079 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mz86w" event={"ID":"91530dee-a144-497b-b63a-3b88b4878905","Type":"ContainerStarted","Data":"458acb3c56b97c48ca9ba842609c3f122a76aeb67f8b53f3e50527beb870e72c"} Sep 19 08:21:37.618218 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.618196 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swsxs" event={"ID":"d4fa1e4d-c5fe-4072-8e44-f40975acb77d","Type":"ContainerStarted","Data":"16d09243e8ae725af0561f3bc0a18899a877d1acb367151c9821b5820053ee07"} Sep 19 08:21:37.619313 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.619292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bd8s" event={"ID":"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449","Type":"ContainerStarted","Data":"33db245ae45d4d6771caf89cfc0814b6574a772e84b2763a6be5e0c8b0f7e3da"} Sep 19 08:21:37.620302 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:37.620277 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l9799" event={"ID":"aa486a4a-823d-40ca-b1d1-df39a7a1baf8","Type":"ContainerStarted","Data":"0960197eab84c8e99160b2eeb3538f6d396dce672268dc5784938c961c3caa4e"} Sep 19 08:21:38.974046 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:38.973544 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:38.974046 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:38.973642 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:38.974708 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:38.974683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37d3bf81-c5c5-4807-8418-ea8dc81793ca-service-ca-bundle\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:38.978043 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:38.978019 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37d3bf81-c5c5-4807-8418-ea8dc81793ca-metrics-certs\") pod \"router-default-56868b9754-pt5rr\" (UID: \"37d3bf81-c5c5-4807-8418-ea8dc81793ca\") " pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:39.167025 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:39.166994 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wc64m\"" Sep 19 08:21:39.175067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:39.175040 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:39.634586 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:39.634484 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bd8s" event={"ID":"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449","Type":"ContainerStarted","Data":"f504321b359573d47a42779bc7e8d7105d6ce99a99619902ace9644bd3edaf27"} Sep 19 08:21:39.834117 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:39.834090 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-56868b9754-pt5rr"] Sep 19 08:21:40.641925 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:40.641884 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swsxs" event={"ID":"d4fa1e4d-c5fe-4072-8e44-f40975acb77d","Type":"ContainerStarted","Data":"d63cf067dddac0a4dfa7ec007b5bd3c8e786c660c0fafca4a5f6926a66379b69"} Sep 19 08:21:40.642316 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:40.641955 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-swsxs" event={"ID":"d4fa1e4d-c5fe-4072-8e44-f40975acb77d","Type":"ContainerStarted","Data":"4d573ab2ecbb2002ceec90a34cfde9d2b681e058a309a02456c4ee1e36d8aa6a"} Sep 19 08:21:40.642316 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:40.641986 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:40.644141 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:40.644117 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bd8s" event={"ID":"30dd1f4f-1ad8-4cca-94f9-ae57d1e43449","Type":"ContainerStarted","Data":"71f580142cedb9243c50df8648c75487765bc6f5992484b7c3e7f7ec6151d886"} Sep 19 08:21:40.659529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:40.659488 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-swsxs" podStartSLOduration=33.65230624 podStartE2EDuration="35.659474487s" totalImagesPullingTime="2.007168247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:40.658726406 +0000 UTC m=+69.870560267" watchObservedRunningTime="2026-09-19 08:21:40.659474487 +0000 UTC m=+69.871308345" Sep 19 08:21:40.675289 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:40.675249 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7bd8s" podStartSLOduration=68.392027987 podStartE2EDuration="1m9.67523287s" totalImagesPullingTime="1.283204883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:40.673757335 +0000 UTC m=+69.885591193" watchObservedRunningTime="2026-09-19 08:21:40.67523287 +0000 UTC m=+69.887066728" Sep 19 08:21:40.829548 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:40.829485 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d3bf81_c5c5_4807_8418_ea8dc81793ca.slice/crio-4643deaa8aeb26470eab909845f06e7c7d47173c1785e670d7362117a2cc6452 WatchSource:0}: Error finding container 4643deaa8aeb26470eab909845f06e7c7d47173c1785e670d7362117a2cc6452: Status 404 returned error can't find the container with id 4643deaa8aeb26470eab909845f06e7c7d47173c1785e670d7362117a2cc6452 Sep 19 08:21:41.651765 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:41.651701 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56868b9754-pt5rr" event={"ID":"37d3bf81-c5c5-4807-8418-ea8dc81793ca","Type":"ContainerStarted","Data":"4643deaa8aeb26470eab909845f06e7c7d47173c1785e670d7362117a2cc6452"} Sep 19 08:21:42.300729 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.300710 2585 scope.go:117] "RemoveContainer" containerID="f6900d30840f9273a2222aaae1b04f5d0c84ce5b1e8c768568cfc99891a1aec9" Sep 19 08:21:42.656193 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.656159 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mz86w" event={"ID":"91530dee-a144-497b-b63a-3b88b4878905","Type":"ContainerStarted","Data":"6b1471e52353031b2cff536adbbb9e6c8f89c4a2824aaa63cb8c23fd51d868ea"} Sep 19 08:21:42.657810 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.657787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-56868b9754-pt5rr" event={"ID":"37d3bf81-c5c5-4807-8418-ea8dc81793ca","Type":"ContainerStarted","Data":"8eed3e059398d223815b0719b085459e70e38649384cd03ce7848757b6978fd9"} Sep 19 08:21:42.659418 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.659397 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:21:42.659897 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.659881 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/1.log" Sep 19 08:21:42.659988 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.659911 2585 generic.go:356] "Generic (PLEG): container finished" podID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" containerID="a8d8b83cc9b2c8ab7d1fa9c4e60f94cbe0af47bc66f2baed260006b21c092630" exitCode=255 Sep 19 08:21:42.659988 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.659952 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" event={"ID":"b88c6ebb-8599-443a-a656-8aa7f8079fd3","Type":"ContainerDied","Data":"a8d8b83cc9b2c8ab7d1fa9c4e60f94cbe0af47bc66f2baed260006b21c092630"} Sep 19 08:21:42.660093 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.660004 2585 scope.go:117] "RemoveContainer" containerID="f6900d30840f9273a2222aaae1b04f5d0c84ce5b1e8c768568cfc99891a1aec9" Sep 19 08:21:42.660271 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.660256 2585 scope.go:117] "RemoveContainer" containerID="a8d8b83cc9b2c8ab7d1fa9c4e60f94cbe0af47bc66f2baed260006b21c092630" Sep 19 08:21:42.660443 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:42.660420 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-nvgxc_openshift-console-operator(b88c6ebb-8599-443a-a656-8aa7f8079fd3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" podUID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" Sep 19 08:21:42.661634 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.661615 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l9799" event={"ID":"aa486a4a-823d-40ca-b1d1-df39a7a1baf8","Type":"ContainerStarted","Data":"311a949369942988bb7017b523ae80154d25e5779b0d8e1179787d8af9d75111"} Sep 19 08:21:42.661743 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.661732 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:21:42.671552 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.671492 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mz86w" podStartSLOduration=66.966125322 podStartE2EDuration="1m11.671480085s" totalImagesPullingTime="4.705354763s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:42.67106472 +0000 UTC m=+71.882898576" watchObservedRunningTime="2026-09-19 08:21:42.671480085 +0000 UTC m=+71.883313941" Sep 19 08:21:42.689874 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.689833 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-56868b9754-pt5rr" podStartSLOduration=35.689819592 podStartE2EDuration="35.689819592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:42.689414921 +0000 UTC m=+71.901248779" watchObservedRunningTime="2026-09-19 08:21:42.689819592 +0000 UTC m=+71.901653450" Sep 19 08:21:42.725535 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:42.725479 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l9799" podStartSLOduration=67.114661711 podStartE2EDuration="1m11.725464948s" totalImagesPullingTime="4.610803237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:42.724908445 +0000 UTC m=+71.936742302" watchObservedRunningTime="2026-09-19 08:21:42.725464948 +0000 UTC m=+71.937298805" Sep 19 08:21:43.175820 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.175785 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:43.178508 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.178482 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:43.614902 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.614803 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:43.617098 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.617070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f32eb384-ceda-409c-be43-f0f8c014541a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fkbpb\" (UID: \"f32eb384-ceda-409c-be43-f0f8c014541a\") " pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:43.666799 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.666766 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:21:43.667347 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.667319 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:43.668372 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.668357 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-56868b9754-pt5rr" Sep 19 08:21:43.791557 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.791511 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jw55w\"" Sep 19 08:21:43.799065 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:43.799038 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fkbpb" Sep 19 08:21:44.110999 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.110970 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fkbpb"] Sep 19 08:21:44.114111 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:44.114084 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32eb384_ceda_409c_be43_f0f8c014541a.slice/crio-dc18f7141aa2e320c97d83fb971db79e9e55074fc3a39bae4f3b2b95364eb43b WatchSource:0}: Error finding container dc18f7141aa2e320c97d83fb971db79e9e55074fc3a39bae4f3b2b95364eb43b: Status 404 returned error can't find the container with id dc18f7141aa2e320c97d83fb971db79e9e55074fc3a39bae4f3b2b95364eb43b Sep 19 08:21:44.420221 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.420193 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn"] Sep 19 08:21:44.445434 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.445399 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.450203 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.450164 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 19 08:21:44.450856 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.450610 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 19 08:21:44.450856 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.450644 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 19 08:21:44.450856 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.450654 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 19 08:21:44.450856 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.450730 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 19 08:21:44.451174 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.451024 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 19 08:21:44.451174 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.451098 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 19 08:21:44.452214 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.452192 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn"] Sep 19 08:21:44.518278 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.518240 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f5796b8ff-crgk7"] Sep 19 08:21:44.520650 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.520630 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.521971 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.521944 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-hub\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.522095 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.522002 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.522095 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.522057 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.522095 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.522088 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5gk2\" (UniqueName: \"kubernetes.io/projected/a184ab23-345f-4b67-97af-c01568bc482e-kube-api-access-r5gk2\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.522236 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.522117 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a184ab23-345f-4b67-97af-c01568bc482e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.522236 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.522145 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-ca\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.523585 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.523480 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 19 08:21:44.524445 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.524423 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 19 08:21:44.524667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.524641 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 19 08:21:44.524990 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.524967 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-8bzsz\"" Sep 19 08:21:44.531175 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.531152 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 19 08:21:44.551653 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.551621 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f5796b8ff-crgk7"] Sep 19 08:21:44.623095 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623059 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-registry-tls\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.623295 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623103 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-hub\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.623295 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-bound-sa-token\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.623295 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623195 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/050777fb-709c-43e8-a675-0f7f24630a43-trusted-ca\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.623295 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623232 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.623295 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.623602 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623459 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/050777fb-709c-43e8-a675-0f7f24630a43-registry-certificates\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.623602 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623502 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5gk2\" (UniqueName: \"kubernetes.io/projected/a184ab23-345f-4b67-97af-c01568bc482e-kube-api-access-r5gk2\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.623602 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623565 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8n7v\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-kube-api-access-w8n7v\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.623602 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623594 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a184ab23-345f-4b67-97af-c01568bc482e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.623916 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623631 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-ca\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.623916 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/050777fb-709c-43e8-a675-0f7f24630a43-ca-trust-extracted\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.623916 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623728 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/050777fb-709c-43e8-a675-0f7f24630a43-installation-pull-secrets\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.623916 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.623884 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/050777fb-709c-43e8-a675-0f7f24630a43-image-registry-private-configuration\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.624445 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.624417 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a184ab23-345f-4b67-97af-c01568bc482e-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.626330 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.626303 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.626454 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.626390 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-ca\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.626517 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.626497 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.626609 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.626588 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a184ab23-345f-4b67-97af-c01568bc482e-hub\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.639426 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.639397 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5gk2\" (UniqueName: \"kubernetes.io/projected/a184ab23-345f-4b67-97af-c01568bc482e-kube-api-access-r5gk2\") pod \"cluster-proxy-proxy-agent-6847d5f749-vzfbn\" (UID: \"a184ab23-345f-4b67-97af-c01568bc482e\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.671858 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.671793 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fkbpb" event={"ID":"f32eb384-ceda-409c-be43-f0f8c014541a","Type":"ContainerStarted","Data":"457b52cdcca75c57cdbbe89a468333081f67655e6e8c47dcc38e4fe02317fa0a"} Sep 19 08:21:44.672234 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.671867 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fkbpb" event={"ID":"f32eb384-ceda-409c-be43-f0f8c014541a","Type":"ContainerStarted","Data":"dc18f7141aa2e320c97d83fb971db79e9e55074fc3a39bae4f3b2b95364eb43b"} Sep 19 08:21:44.725175 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725144 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/050777fb-709c-43e8-a675-0f7f24630a43-ca-trust-extracted\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.725265 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725182 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/050777fb-709c-43e8-a675-0f7f24630a43-installation-pull-secrets\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.725265 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725211 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/050777fb-709c-43e8-a675-0f7f24630a43-image-registry-private-configuration\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.725265 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725249 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-registry-tls\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.725410 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-bound-sa-token\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.725598 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725548 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/050777fb-709c-43e8-a675-0f7f24630a43-trusted-ca\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.725598 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725569 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/050777fb-709c-43e8-a675-0f7f24630a43-ca-trust-extracted\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.725744 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725632 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/050777fb-709c-43e8-a675-0f7f24630a43-registry-certificates\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.725744 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.725691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8n7v\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-kube-api-access-w8n7v\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.726436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.726402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/050777fb-709c-43e8-a675-0f7f24630a43-registry-certificates\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.726556 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.726513 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/050777fb-709c-43e8-a675-0f7f24630a43-trusted-ca\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.727786 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.727761 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/050777fb-709c-43e8-a675-0f7f24630a43-image-registry-private-configuration\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.728031 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.728004 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/050777fb-709c-43e8-a675-0f7f24630a43-installation-pull-secrets\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.728381 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.728362 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-registry-tls\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.744922 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.744898 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-bound-sa-token\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.754378 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.754351 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8n7v\" (UniqueName: \"kubernetes.io/projected/050777fb-709c-43e8-a675-0f7f24630a43-kube-api-access-w8n7v\") pod \"image-registry-5f5796b8ff-crgk7\" (UID: \"050777fb-709c-43e8-a675-0f7f24630a43\") " pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:44.772251 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.772230 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" Sep 19 08:21:44.832255 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:44.832222 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:45.045919 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:45.045895 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn"] Sep 19 08:21:45.048194 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:45.048166 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda184ab23_345f_4b67_97af_c01568bc482e.slice/crio-a26ca66fd4fa737dec9a3d012d6996a98300124c16a93a6a97379543cea2d27b WatchSource:0}: Error finding container a26ca66fd4fa737dec9a3d012d6996a98300124c16a93a6a97379543cea2d27b: Status 404 returned error can't find the container with id a26ca66fd4fa737dec9a3d012d6996a98300124c16a93a6a97379543cea2d27b Sep 19 08:21:45.105708 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:45.105568 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f5796b8ff-crgk7"] Sep 19 08:21:45.108281 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:45.108244 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod050777fb_709c_43e8_a675_0f7f24630a43.slice/crio-48539f0de6bd8a64c45e6f7c47b90e9e49274778efb8e50d0f888a39da6a39a4 WatchSource:0}: Error finding container 48539f0de6bd8a64c45e6f7c47b90e9e49274778efb8e50d0f888a39da6a39a4: Status 404 returned error can't find the container with id 48539f0de6bd8a64c45e6f7c47b90e9e49274778efb8e50d0f888a39da6a39a4 Sep 19 08:21:45.677909 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:45.677867 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fkbpb" event={"ID":"f32eb384-ceda-409c-be43-f0f8c014541a","Type":"ContainerStarted","Data":"7dcbc79ba2bf71051aee260705312cf9c31a178be5c65ef3cf06f3bd8e9e12ca"} Sep 19 08:21:45.679273 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:45.679239 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" event={"ID":"a184ab23-345f-4b67-97af-c01568bc482e","Type":"ContainerStarted","Data":"a26ca66fd4fa737dec9a3d012d6996a98300124c16a93a6a97379543cea2d27b"} Sep 19 08:21:45.680882 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:45.680855 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" event={"ID":"050777fb-709c-43e8-a675-0f7f24630a43","Type":"ContainerStarted","Data":"0be927d53c48830bf1570b4990421e2e01747d5f6da6ba56a559fdc8700b846d"} Sep 19 08:21:45.680966 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:45.680901 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" event={"ID":"050777fb-709c-43e8-a675-0f7f24630a43","Type":"ContainerStarted","Data":"48539f0de6bd8a64c45e6f7c47b90e9e49274778efb8e50d0f888a39da6a39a4"} Sep 19 08:21:45.701575 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:45.701506 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" podStartSLOduration=1.7014888419999998 podStartE2EDuration="1.701488842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:45.700441742 +0000 UTC m=+74.912275599" watchObservedRunningTime="2026-09-19 08:21:45.701488842 +0000 UTC m=+74.913322697" Sep 19 08:21:46.687173 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:46.687134 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fkbpb" event={"ID":"f32eb384-ceda-409c-be43-f0f8c014541a","Type":"ContainerStarted","Data":"7f945bd50bd60a5a0adaa179af100d86bf12031d09991793e7d6a8f12c912a26"} Sep 19 08:21:46.687689 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:46.687338 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:21:46.711244 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:46.711195 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fkbpb" podStartSLOduration=34.078262163 podStartE2EDuration="35.711179003s" totalImagesPullingTime="1.63291684s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 08:21:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:46.710503474 +0000 UTC m=+75.922337345" watchObservedRunningTime="2026-09-19 08:21:46.711179003 +0000 UTC m=+75.923012859" Sep 19 08:21:47.423658 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:47.423625 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:47.424028 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:47.423720 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:21:47.424302 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:47.424124 2585 scope.go:117] "RemoveContainer" containerID="a8d8b83cc9b2c8ab7d1fa9c4e60f94cbe0af47bc66f2baed260006b21c092630" Sep 19 08:21:47.424429 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:47.424308 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-nvgxc_openshift-console-operator(b88c6ebb-8599-443a-a656-8aa7f8079fd3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" podUID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" Sep 19 08:21:49.700101 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:49.700058 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" event={"ID":"a184ab23-345f-4b67-97af-c01568bc482e","Type":"ContainerStarted","Data":"d033e2a6141d534d6fc91d6fb14bfeedc1b4ebe75463bd317a50a48510094e41"} Sep 19 08:21:49.700601 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:49.700116 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" event={"ID":"a184ab23-345f-4b67-97af-c01568bc482e","Type":"ContainerStarted","Data":"358ea911631bbac5369c3d45623a35979e060583583f12c27c36ee5d87f0a039"} Sep 19 08:21:49.700601 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:49.700135 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" event={"ID":"a184ab23-345f-4b67-97af-c01568bc482e","Type":"ContainerStarted","Data":"baa121067d041149841ee742b21c095a911b30d5f76facb675a49cd2d3e86486"} Sep 19 08:21:49.723413 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:49.723367 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6847d5f749-vzfbn" podStartSLOduration=1.909989987 podStartE2EDuration="5.723350638s" totalImagesPullingTime="3.813360651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:49.721723404 +0000 UTC m=+78.933557261" watchObservedRunningTime="2026-09-19 08:21:49.723350638 +0000 UTC m=+78.935184495" Sep 19 08:21:50.654325 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:50.654291 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-swsxs" Sep 19 08:21:52.809945 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.809906 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mx8mc"] Sep 19 08:21:52.813407 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.813390 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.816561 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.816513 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 19 08:21:52.816704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.816654 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 19 08:21:52.816953 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.816939 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 19 08:21:52.817024 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.817011 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 19 08:21:52.817849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.817831 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 19 08:21:52.817954 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.817892 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-f7j7t\"" Sep 19 08:21:52.818021 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.817964 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 19 08:21:52.895197 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895165 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.895197 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895199 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.895390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-sys\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.895390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895282 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-textfile\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.895390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895321 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-metrics-client-ca\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.895390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895338 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-root\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.895390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895353 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-tls\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.895612 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895408 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwtg\" (UniqueName: \"kubernetes.io/projected/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-kube-api-access-szwtg\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.895612 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.895432 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-wtmp\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996000 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.995970 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-metrics-client-ca\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996000 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996000 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-root\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996184 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-tls\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996184 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996048 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwtg\" (UniqueName: \"kubernetes.io/projected/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-kube-api-access-szwtg\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996184 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-wtmp\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996184 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-root\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996184 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996098 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996184 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996155 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996267 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-sys\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996281 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-wtmp\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996320 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-textfile\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996389 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-sys\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996688 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996670 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-metrics-client-ca\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.996746 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.996708 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-textfile\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.997369 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.997346 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-accelerators-collector-config\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.998389 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.998361 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-tls\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:52.998561 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:52.998534 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:53.005915 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:53.005888 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwtg\" (UniqueName: \"kubernetes.io/projected/3efd392b-dc87-4b0d-baaf-8b88dc2bc206-kube-api-access-szwtg\") pod \"node-exporter-mx8mc\" (UID: \"3efd392b-dc87-4b0d-baaf-8b88dc2bc206\") " pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:53.122649 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:53.122586 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mx8mc" Sep 19 08:21:53.133691 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:21:53.133656 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3efd392b_dc87_4b0d_baaf_8b88dc2bc206.slice/crio-2e1d616dab2a137702b67ba35aa63f17d16af069299e23aced54f4f91184937c WatchSource:0}: Error finding container 2e1d616dab2a137702b67ba35aa63f17d16af069299e23aced54f4f91184937c: Status 404 returned error can't find the container with id 2e1d616dab2a137702b67ba35aa63f17d16af069299e23aced54f4f91184937c Sep 19 08:21:53.714793 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:53.714646 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx8mc" event={"ID":"3efd392b-dc87-4b0d-baaf-8b88dc2bc206","Type":"ContainerStarted","Data":"2e1d616dab2a137702b67ba35aa63f17d16af069299e23aced54f4f91184937c"} Sep 19 08:21:54.720061 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:54.720021 2585 generic.go:356] "Generic (PLEG): container finished" podID="3efd392b-dc87-4b0d-baaf-8b88dc2bc206" containerID="6c7dc28bcf648aa868b0e808d6da30a36822ceea311df4f953caa305e5a0b58b" exitCode=0 Sep 19 08:21:54.720441 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:54.720108 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx8mc" event={"ID":"3efd392b-dc87-4b0d-baaf-8b88dc2bc206","Type":"ContainerDied","Data":"6c7dc28bcf648aa868b0e808d6da30a36822ceea311df4f953caa305e5a0b58b"} Sep 19 08:21:55.726373 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:55.726337 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx8mc" event={"ID":"3efd392b-dc87-4b0d-baaf-8b88dc2bc206","Type":"ContainerStarted","Data":"e1c8a6f4c4a36950019e55be3fb836acc12f830be44a23fcdc13f38c7419db2d"} Sep 19 08:21:55.726754 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:55.726394 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mx8mc" event={"ID":"3efd392b-dc87-4b0d-baaf-8b88dc2bc206","Type":"ContainerStarted","Data":"9ba44c9f1e68358b5ecb84691dbc464e61e207821137fb8a9881353d3f5fffdf"} Sep 19 08:21:55.751550 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:55.751493 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mx8mc" podStartSLOduration=2.717252718 podStartE2EDuration="3.751478986s" totalImagesPullingTime="973.407609ms" totalInitContainerRuntime="60.818659ms" isStatefulPod=true podCreationTimestamp="2026-09-19 08:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:21:55.750178157 +0000 UTC m=+84.962012007" watchObservedRunningTime="2026-09-19 08:21:55.751478986 +0000 UTC m=+84.963312843" Sep 19 08:21:58.300772 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:21:58.300739 2585 scope.go:117] "RemoveContainer" containerID="a8d8b83cc9b2c8ab7d1fa9c4e60f94cbe0af47bc66f2baed260006b21c092630" Sep 19 08:21:58.301149 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:21:58.300915 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-fd4bb46c-nvgxc_openshift-console-operator(b88c6ebb-8599-443a-a656-8aa7f8079fd3)\"" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" podUID="b88c6ebb-8599-443a-a656-8aa7f8079fd3" Sep 19 08:22:07.695944 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:07.695915 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f5796b8ff-crgk7" Sep 19 08:22:12.301707 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:12.301672 2585 scope.go:117] "RemoveContainer" containerID="a8d8b83cc9b2c8ab7d1fa9c4e60f94cbe0af47bc66f2baed260006b21c092630" Sep 19 08:22:12.785401 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:12.785370 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:22:12.785594 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:12.785449 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" event={"ID":"b88c6ebb-8599-443a-a656-8aa7f8079fd3","Type":"ContainerStarted","Data":"0d6d392b0297c636e4a5336dda96c94e51bb3ba8abd30a0f7688ff8495af2106"} Sep 19 08:22:12.785764 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:12.785745 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:22:12.822590 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:12.822541 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" podStartSLOduration=52.995018868 podStartE2EDuration="55.822514271s" totalImagesPullingTime="2.827495403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:22:12.820455735 +0000 UTC m=+102.032289593" watchObservedRunningTime="2026-09-19 08:22:12.822514271 +0000 UTC m=+102.034348187" Sep 19 08:22:13.045800 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:13.045731 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-fd4bb46c-nvgxc" Sep 19 08:22:13.669767 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:13.669739 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l9799" Sep 19 08:22:15.798236 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:15.798202 2585 generic.go:356] "Generic (PLEG): container finished" podID="b1e03626-0f6b-41e4-a0c3-9e19c2d2de72" containerID="905299bf12fe34ef198e74a5fb1022ff61dab2c9ff911d8be93ff0d08f336c87" exitCode=0 Sep 19 08:22:15.798571 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:15.798283 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-268wh" event={"ID":"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72","Type":"ContainerDied","Data":"905299bf12fe34ef198e74a5fb1022ff61dab2c9ff911d8be93ff0d08f336c87"} Sep 19 08:22:15.798628 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:15.798612 2585 scope.go:117] "RemoveContainer" containerID="905299bf12fe34ef198e74a5fb1022ff61dab2c9ff911d8be93ff0d08f336c87" Sep 19 08:22:16.805019 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:16.804990 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-6f9c644dd9-268wh" event={"ID":"b1e03626-0f6b-41e4-a0c3-9e19c2d2de72","Type":"ContainerStarted","Data":"198317867c38d6b9a33aa413e6aff4ac7de0a0f039ef19d391a5889a338ea4d7"} Sep 19 08:22:16.960723 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:16.960698 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56868b9754-pt5rr_37d3bf81-c5c5-4807-8418-ea8dc81793ca/router/0.log" Sep 19 08:22:16.978965 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:16.978936 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v68l8_9cd4ad62-dadf-42d7-8696-93041ca4dd12/serve-healthcheck-canary/0.log" Sep 19 08:22:46.918916 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:46.918882 2585 generic.go:356] "Generic (PLEG): container finished" podID="10f7eb02-068f-444d-8695-17e0d5f7609e" containerID="a551002591750e260bc9f996c6dddfe2e3c6524dc3fa1b53ae5f0b1475b397cd" exitCode=0 Sep 19 08:22:46.919301 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:46.918931 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" event={"ID":"10f7eb02-068f-444d-8695-17e0d5f7609e","Type":"ContainerDied","Data":"a551002591750e260bc9f996c6dddfe2e3c6524dc3fa1b53ae5f0b1475b397cd"} Sep 19 08:22:46.919301 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:46.919225 2585 scope.go:117] "RemoveContainer" containerID="a551002591750e260bc9f996c6dddfe2e3c6524dc3fa1b53ae5f0b1475b397cd" Sep 19 08:22:47.923775 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:22:47.923736 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-2tss2" event={"ID":"10f7eb02-068f-444d-8695-17e0d5f7609e","Type":"ContainerStarted","Data":"946a5a6ade2cacf56908fe7dd7de8b4eee58681a53560fd11007655333a83e2a"} Sep 19 08:25:31.157458 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:25:31.157422 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:25:31.160849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:25:31.160822 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:25:31.168822 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:25:31.168796 2585 kubelet.go:1648] "Image garbage collection succeeded" Sep 19 08:27:27.709993 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.709962 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-d45vs"] Sep 19 08:27:27.712869 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.712851 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:27.715474 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.715435 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-rbdpz\"" Sep 19 08:27:27.715474 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.715456 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 19 08:27:27.715646 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.715486 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 19 08:27:27.716506 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.716488 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 19 08:27:27.721765 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.721744 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d45vs"] Sep 19 08:27:27.804799 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.804765 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fce49e3c-bddd-4665-a410-5f8fcaca987c-data\") pod \"seaweedfs-64568bcd49-d45vs\" (UID: \"fce49e3c-bddd-4665-a410-5f8fcaca987c\") " pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:27.804949 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.804819 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt62k\" (UniqueName: \"kubernetes.io/projected/fce49e3c-bddd-4665-a410-5f8fcaca987c-kube-api-access-dt62k\") pod \"seaweedfs-64568bcd49-d45vs\" (UID: \"fce49e3c-bddd-4665-a410-5f8fcaca987c\") " pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:27.905823 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.905795 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt62k\" (UniqueName: \"kubernetes.io/projected/fce49e3c-bddd-4665-a410-5f8fcaca987c-kube-api-access-dt62k\") pod \"seaweedfs-64568bcd49-d45vs\" (UID: \"fce49e3c-bddd-4665-a410-5f8fcaca987c\") " pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:27.905990 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.905844 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fce49e3c-bddd-4665-a410-5f8fcaca987c-data\") pod \"seaweedfs-64568bcd49-d45vs\" (UID: \"fce49e3c-bddd-4665-a410-5f8fcaca987c\") " pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:27.906162 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.906146 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/fce49e3c-bddd-4665-a410-5f8fcaca987c-data\") pod \"seaweedfs-64568bcd49-d45vs\" (UID: \"fce49e3c-bddd-4665-a410-5f8fcaca987c\") " pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:27.918926 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:27.918903 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt62k\" (UniqueName: \"kubernetes.io/projected/fce49e3c-bddd-4665-a410-5f8fcaca987c-kube-api-access-dt62k\") pod \"seaweedfs-64568bcd49-d45vs\" (UID: \"fce49e3c-bddd-4665-a410-5f8fcaca987c\") " pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:28.021956 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:28.021860 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:28.290199 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:28.290164 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-d45vs"] Sep 19 08:27:28.292788 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:27:28.292758 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce49e3c_bddd_4665_a410_5f8fcaca987c.slice/crio-bc73c15290f1f48efcc3d821da48a600ae05d8325044c4a99e49548ea2e2f265 WatchSource:0}: Error finding container bc73c15290f1f48efcc3d821da48a600ae05d8325044c4a99e49548ea2e2f265: Status 404 returned error can't find the container with id bc73c15290f1f48efcc3d821da48a600ae05d8325044c4a99e49548ea2e2f265 Sep 19 08:27:28.294022 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:28.293999 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 19 08:27:28.894887 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:28.894847 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d45vs" event={"ID":"fce49e3c-bddd-4665-a410-5f8fcaca987c","Type":"ContainerStarted","Data":"bc73c15290f1f48efcc3d821da48a600ae05d8325044c4a99e49548ea2e2f265"} Sep 19 08:27:31.476473 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:31.476444 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 19 08:27:31.908182 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:31.908140 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-d45vs" event={"ID":"fce49e3c-bddd-4665-a410-5f8fcaca987c","Type":"ContainerStarted","Data":"4b33d3bdd3d64396d60ceff5c9baefe53829f0ad56022893bb6475d5309fbd25"} Sep 19 08:27:31.908350 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:31.908213 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:27:31.924689 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:31.924610 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-d45vs" podStartSLOduration=1.744807285 podStartE2EDuration="4.924578721s" totalImagesPullingTime="3.179771436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:27:31.923778473 +0000 UTC m=+421.135612331" watchObservedRunningTime="2026-09-19 08:27:31.924578721 +0000 UTC m=+421.136412578" Sep 19 08:27:37.914542 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:27:37.914496 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-d45vs" Sep 19 08:28:44.315976 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.315945 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-b2vls"] Sep 19 08:28:44.319144 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.319126 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:44.321655 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.321629 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-6dklw\"" Sep 19 08:28:44.321772 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.321651 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\"" Sep 19 08:28:44.329093 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.329068 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-b2vls"] Sep 19 08:28:44.457277 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.457239 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk7t5\" (UniqueName: \"kubernetes.io/projected/73c79571-bb8e-4e1b-b766-2656b9970517-kube-api-access-nk7t5\") pod \"model-serving-api-7b954ffc95-b2vls\" (UID: \"73c79571-bb8e-4e1b-b766-2656b9970517\") " pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:44.457469 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.457296 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c79571-bb8e-4e1b-b766-2656b9970517-tls-certs\") pod \"model-serving-api-7b954ffc95-b2vls\" (UID: \"73c79571-bb8e-4e1b-b766-2656b9970517\") " pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:44.558722 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.558680 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c79571-bb8e-4e1b-b766-2656b9970517-tls-certs\") pod \"model-serving-api-7b954ffc95-b2vls\" (UID: \"73c79571-bb8e-4e1b-b766-2656b9970517\") " pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:44.558916 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.558771 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk7t5\" (UniqueName: \"kubernetes.io/projected/73c79571-bb8e-4e1b-b766-2656b9970517-kube-api-access-nk7t5\") pod \"model-serving-api-7b954ffc95-b2vls\" (UID: \"73c79571-bb8e-4e1b-b766-2656b9970517\") " pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:44.558916 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:28:44.558825 2585 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 19 08:28:44.558916 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:28:44.558913 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/73c79571-bb8e-4e1b-b766-2656b9970517-tls-certs podName:73c79571-bb8e-4e1b-b766-2656b9970517 nodeName:}" failed. No retries permitted until 2026-09-19 08:28:45.058896355 +0000 UTC m=+494.270730190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/73c79571-bb8e-4e1b-b766-2656b9970517-tls-certs") pod "model-serving-api-7b954ffc95-b2vls" (UID: "73c79571-bb8e-4e1b-b766-2656b9970517") : secret "model-serving-api-tls" not found Sep 19 08:28:44.568237 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:44.568171 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk7t5\" (UniqueName: \"kubernetes.io/projected/73c79571-bb8e-4e1b-b766-2656b9970517-kube-api-access-nk7t5\") pod \"model-serving-api-7b954ffc95-b2vls\" (UID: \"73c79571-bb8e-4e1b-b766-2656b9970517\") " pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:45.062169 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:45.062129 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c79571-bb8e-4e1b-b766-2656b9970517-tls-certs\") pod \"model-serving-api-7b954ffc95-b2vls\" (UID: \"73c79571-bb8e-4e1b-b766-2656b9970517\") " pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:45.064516 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:45.064489 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c79571-bb8e-4e1b-b766-2656b9970517-tls-certs\") pod \"model-serving-api-7b954ffc95-b2vls\" (UID: \"73c79571-bb8e-4e1b-b766-2656b9970517\") " pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:45.229847 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:45.229812 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:45.497711 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:45.497675 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-b2vls"] Sep 19 08:28:45.501065 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:28:45.501037 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c79571_bb8e_4e1b_b766_2656b9970517.slice/crio-be4de71e3bb11a8f6307f5ee08f68009e7e9f2b414e0f37eaf6e4f621b307332 WatchSource:0}: Error finding container be4de71e3bb11a8f6307f5ee08f68009e7e9f2b414e0f37eaf6e4f621b307332: Status 404 returned error can't find the container with id be4de71e3bb11a8f6307f5ee08f68009e7e9f2b414e0f37eaf6e4f621b307332 Sep 19 08:28:46.168745 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:46.168667 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-b2vls" event={"ID":"73c79571-bb8e-4e1b-b766-2656b9970517","Type":"ContainerStarted","Data":"be4de71e3bb11a8f6307f5ee08f68009e7e9f2b414e0f37eaf6e4f621b307332"} Sep 19 08:28:48.177324 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:48.177282 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-b2vls" event={"ID":"73c79571-bb8e-4e1b-b766-2656b9970517","Type":"ContainerStarted","Data":"41dba3e877caaf6fc31b9fbcbdadbd7b83bf573e6e08a1356468aee241994605"} Sep 19 08:28:48.177834 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:48.177410 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:28:48.196141 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:48.196092 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-b2vls" podStartSLOduration=1.957331435 podStartE2EDuration="4.196074052s" totalImagesPullingTime="2.238742617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:28:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:28:48.194548777 +0000 UTC m=+497.406382631" watchObservedRunningTime="2026-09-19 08:28:48.196074052 +0000 UTC m=+497.407907905" Sep 19 08:28:59.185807 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:28:59.185778 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-b2vls" Sep 19 08:30:31.182341 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.182313 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:30:31.182937 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.182919 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:30:31.384509 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.384476 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j"] Sep 19 08:30:31.392634 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.392609 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:31.395111 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.395079 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-83f4f-predictor-serving-cert\"" Sep 19 08:30:31.395266 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.395239 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 19 08:30:31.396363 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.396339 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-83f4f-kube-rbac-proxy-sar-config\"" Sep 19 08:30:31.396470 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.396415 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 19 08:30:31.396470 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.396416 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-zd5sl\"" Sep 19 08:30:31.398184 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.398164 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j"] Sep 19 08:30:31.491963 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.491870 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9crt\" (UniqueName: \"kubernetes.io/projected/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-api-access-q9crt\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:31.491963 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.491920 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:31.492144 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.491996 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:31.592424 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.592386 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:31.592424 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.592438 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9crt\" (UniqueName: \"kubernetes.io/projected/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-api-access-q9crt\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:31.592674 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.592477 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:31.592674 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:30:31.592545 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-serving-cert: secret "success-200-isvc-83f4f-predictor-serving-cert" not found Sep 19 08:30:31.592674 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:30:31.592621 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls podName:f0b36ea0-f156-4b94-9be4-fde292f1a18a nodeName:}" failed. No retries permitted until 2026-09-19 08:30:32.092603607 +0000 UTC m=+601.304437459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls") pod "success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" (UID: "f0b36ea0-f156-4b94-9be4-fde292f1a18a") : secret "success-200-isvc-83f4f-predictor-serving-cert" not found Sep 19 08:30:31.593125 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.593105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:31.602374 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:31.602346 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9crt\" (UniqueName: \"kubernetes.io/projected/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-api-access-q9crt\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:32.097094 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:32.097053 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:32.099513 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:32.099487 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls\") pod \"success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:32.303957 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:32.303918 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:32.575162 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:32.575139 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j"] Sep 19 08:30:32.577614 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:30:32.577584 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b36ea0_f156_4b94_9be4_fde292f1a18a.slice/crio-e9047893659a8e2b790226f952cce36dfc29c02bd00a0e2394185df2cca590eb WatchSource:0}: Error finding container e9047893659a8e2b790226f952cce36dfc29c02bd00a0e2394185df2cca590eb: Status 404 returned error can't find the container with id e9047893659a8e2b790226f952cce36dfc29c02bd00a0e2394185df2cca590eb Sep 19 08:30:33.550283 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:33.550238 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" event={"ID":"f0b36ea0-f156-4b94-9be4-fde292f1a18a","Type":"ContainerStarted","Data":"e9047893659a8e2b790226f952cce36dfc29c02bd00a0e2394185df2cca590eb"} Sep 19 08:30:44.602135 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:44.602047 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" event={"ID":"f0b36ea0-f156-4b94-9be4-fde292f1a18a","Type":"ContainerStarted","Data":"0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f"} Sep 19 08:30:47.616060 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:47.616018 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" event={"ID":"f0b36ea0-f156-4b94-9be4-fde292f1a18a","Type":"ContainerStarted","Data":"d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919"} Sep 19 08:30:47.616581 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:47.616222 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:47.616581 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:47.616352 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:47.617768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:47.617735 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" Sep 19 08:30:47.635509 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:47.635459 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podStartSLOduration=2.622698877 podStartE2EDuration="16.63544331s" totalImagesPullingTime="14.012744433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:30:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:30:47.633217878 +0000 UTC m=+616.845051774" watchObservedRunningTime="2026-09-19 08:30:47.63544331 +0000 UTC m=+616.847277166" Sep 19 08:30:48.620889 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:48.620848 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" Sep 19 08:30:53.625096 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:53.625026 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:30:53.625604 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:30:53.625577 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" Sep 19 08:31:03.625951 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:31:03.625912 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" Sep 19 08:31:13.625918 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:31:13.625877 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" Sep 19 08:31:23.625743 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:31:23.625697 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" Sep 19 08:31:33.626309 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:31:33.626269 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" Sep 19 08:31:43.626678 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:31:43.626640 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:32:01.472534 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.472492 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb"] Sep 19 08:32:01.475632 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.475607 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:01.478031 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.478003 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-83f4f-serving-cert\"" Sep 19 08:32:01.478130 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.478104 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-83f4f-kube-rbac-proxy-sar-config\"" Sep 19 08:32:01.485177 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.485146 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb"] Sep 19 08:32:01.596661 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.596613 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls\") pod \"switch-graph-83f4f-564cdf5744-btvcb\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:01.596828 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.596696 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-openshift-service-ca-bundle\") pod \"switch-graph-83f4f-564cdf5744-btvcb\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:01.697750 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.697701 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls\") pod \"switch-graph-83f4f-564cdf5744-btvcb\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:01.697924 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.697766 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-openshift-service-ca-bundle\") pod \"switch-graph-83f4f-564cdf5744-btvcb\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:01.697924 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:01.697866 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-83f4f-serving-cert: secret "switch-graph-83f4f-serving-cert" not found Sep 19 08:32:01.698000 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:01.697943 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls podName:aa193bdf-6ea6-4cb7-a374-b7db1d496f17 nodeName:}" failed. No retries permitted until 2026-09-19 08:32:02.197924813 +0000 UTC m=+691.409758660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls") pod "switch-graph-83f4f-564cdf5744-btvcb" (UID: "aa193bdf-6ea6-4cb7-a374-b7db1d496f17") : secret "switch-graph-83f4f-serving-cert" not found Sep 19 08:32:01.698352 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:01.698335 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-openshift-service-ca-bundle\") pod \"switch-graph-83f4f-564cdf5744-btvcb\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:02.202320 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:02.202271 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls\") pod \"switch-graph-83f4f-564cdf5744-btvcb\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:02.204750 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:02.204726 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls\") pod \"switch-graph-83f4f-564cdf5744-btvcb\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:02.385067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:02.385032 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:02.654252 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:02.654230 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb"] Sep 19 08:32:02.656501 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:32:02.656474 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa193bdf_6ea6_4cb7_a374_b7db1d496f17.slice/crio-8e934a049e12f933dbd9a605ff83465319df9421e06579a6cefea650bc4d1e01 WatchSource:0}: Error finding container 8e934a049e12f933dbd9a605ff83465319df9421e06579a6cefea650bc4d1e01: Status 404 returned error can't find the container with id 8e934a049e12f933dbd9a605ff83465319df9421e06579a6cefea650bc4d1e01 Sep 19 08:32:02.889896 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:02.889862 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" event={"ID":"aa193bdf-6ea6-4cb7-a374-b7db1d496f17","Type":"ContainerStarted","Data":"8e934a049e12f933dbd9a605ff83465319df9421e06579a6cefea650bc4d1e01"} Sep 19 08:32:04.898662 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:04.898626 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" event={"ID":"aa193bdf-6ea6-4cb7-a374-b7db1d496f17","Type":"ContainerStarted","Data":"c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d"} Sep 19 08:32:04.899010 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:04.898697 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:04.915563 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:04.915499 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" podStartSLOduration=1.808013242 podStartE2EDuration="3.91548656s" totalImagesPullingTime="2.107473318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:32:04.914041293 +0000 UTC m=+694.125875149" watchObservedRunningTime="2026-09-19 08:32:04.91548656 +0000 UTC m=+694.127320417" Sep 19 08:32:10.908070 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:10.908043 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:12.451006 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.450969 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb"] Sep 19 08:32:12.451382 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.451236 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" containerID="cri-o://c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d" gracePeriod=30 Sep 19 08:32:12.573753 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.573722 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j"] Sep 19 08:32:12.573993 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.573944 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" containerID="cri-o://0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f" gracePeriod=30 Sep 19 08:32:12.574082 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.573975 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kube-rbac-proxy" containerID="cri-o://d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919" gracePeriod=30 Sep 19 08:32:12.718026 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.717944 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6"] Sep 19 08:32:12.721128 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.721112 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.723679 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.723646 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-96102-kube-rbac-proxy-sar-config\"" Sep 19 08:32:12.723679 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.723646 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-96102-predictor-serving-cert\"" Sep 19 08:32:12.731330 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.731306 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6"] Sep 19 08:32:12.789214 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.789183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.789400 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.789226 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.789400 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.789318 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r8hk\" (UniqueName: \"kubernetes.io/projected/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-api-access-9r8hk\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.890477 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.890433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.890697 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.890491 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.890697 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.890590 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r8hk\" (UniqueName: \"kubernetes.io/projected/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-api-access-9r8hk\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.890697 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:12.890678 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-96102-predictor-serving-cert: secret "success-200-isvc-96102-predictor-serving-cert" not found Sep 19 08:32:12.890848 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:12.890736 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls podName:ee89b915-2e78-4f8d-8773-770ba9eb6d4c nodeName:}" failed. No retries permitted until 2026-09-19 08:32:13.390720876 +0000 UTC m=+702.602554716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls") pod "success-200-isvc-96102-predictor-6db489744f-tmcb6" (UID: "ee89b915-2e78-4f8d-8773-770ba9eb6d4c") : secret "success-200-isvc-96102-predictor-serving-cert" not found Sep 19 08:32:12.891110 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.891089 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.899143 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.899122 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r8hk\" (UniqueName: \"kubernetes.io/projected/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-api-access-9r8hk\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:12.929401 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.929377 2585 generic.go:356] "Generic (PLEG): container finished" podID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerID="d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919" exitCode=2 Sep 19 08:32:12.929549 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:12.929462 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" event={"ID":"f0b36ea0-f156-4b94-9be4-fde292f1a18a","Type":"ContainerDied","Data":"d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919"} Sep 19 08:32:13.395104 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:13.395071 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:13.397461 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:13.397438 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls\") pod \"success-200-isvc-96102-predictor-6db489744f-tmcb6\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:13.621616 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:13.621574 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.19:8643/healthz\": dial tcp 10.133.0.19:8643: connect: connection refused" Sep 19 08:32:13.626308 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:13.626281 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.19:8080: connect: connection refused" Sep 19 08:32:13.631644 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:13.631619 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:13.901870 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:13.901842 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6"] Sep 19 08:32:13.904344 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:32:13.904318 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee89b915_2e78_4f8d_8773_770ba9eb6d4c.slice/crio-39191feea04e10bceacc0f72b19fea426998b22491ff4e6a04655fdfb21e8d7e WatchSource:0}: Error finding container 39191feea04e10bceacc0f72b19fea426998b22491ff4e6a04655fdfb21e8d7e: Status 404 returned error can't find the container with id 39191feea04e10bceacc0f72b19fea426998b22491ff4e6a04655fdfb21e8d7e Sep 19 08:32:13.934340 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:13.934280 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" event={"ID":"ee89b915-2e78-4f8d-8773-770ba9eb6d4c","Type":"ContainerStarted","Data":"39191feea04e10bceacc0f72b19fea426998b22491ff4e6a04655fdfb21e8d7e"} Sep 19 08:32:14.940277 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:14.940239 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" event={"ID":"ee89b915-2e78-4f8d-8773-770ba9eb6d4c","Type":"ContainerStarted","Data":"60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495"} Sep 19 08:32:14.940680 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:14.940292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" event={"ID":"ee89b915-2e78-4f8d-8773-770ba9eb6d4c","Type":"ContainerStarted","Data":"97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404"} Sep 19 08:32:14.940680 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:14.940413 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:14.959203 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:14.959156 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podStartSLOduration=2.959141098 podStartE2EDuration="2.959141098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:32:14.958433944 +0000 UTC m=+704.170267801" watchObservedRunningTime="2026-09-19 08:32:14.959141098 +0000 UTC m=+704.170974955" Sep 19 08:32:15.906313 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:15.906279 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:32:15.949920 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:15.949870 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:15.951345 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:15.951315 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" Sep 19 08:32:16.121582 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.121557 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:32:16.216779 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.216737 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls\") pod \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " Sep 19 08:32:16.216779 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.216786 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9crt\" (UniqueName: \"kubernetes.io/projected/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-api-access-q9crt\") pod \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " Sep 19 08:32:16.217286 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.216866 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-rbac-proxy-sar-config\") pod \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\" (UID: \"f0b36ea0-f156-4b94-9be4-fde292f1a18a\") " Sep 19 08:32:16.217286 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.217206 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f0b36ea0-f156-4b94-9be4-fde292f1a18a" (UID: "f0b36ea0-f156-4b94-9be4-fde292f1a18a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:32:16.218986 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.218964 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f0b36ea0-f156-4b94-9be4-fde292f1a18a" (UID: "f0b36ea0-f156-4b94-9be4-fde292f1a18a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:32:16.219069 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.219017 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-api-access-q9crt" (OuterVolumeSpecName: "kube-api-access-q9crt") pod "f0b36ea0-f156-4b94-9be4-fde292f1a18a" (UID: "f0b36ea0-f156-4b94-9be4-fde292f1a18a"). InnerVolumeSpecName "kube-api-access-q9crt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 08:32:16.317682 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.317648 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f0b36ea0-f156-4b94-9be4-fde292f1a18a-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:32:16.317682 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.317676 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9crt\" (UniqueName: \"kubernetes.io/projected/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-api-access-q9crt\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:32:16.317682 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.317686 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f0b36ea0-f156-4b94-9be4-fde292f1a18a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:32:16.954454 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.954419 2585 generic.go:356] "Generic (PLEG): container finished" podID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerID="0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f" exitCode=0 Sep 19 08:32:16.954863 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.954470 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" Sep 19 08:32:16.954863 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.954502 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" event={"ID":"f0b36ea0-f156-4b94-9be4-fde292f1a18a","Type":"ContainerDied","Data":"0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f"} Sep 19 08:32:16.954863 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.954570 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j" event={"ID":"f0b36ea0-f156-4b94-9be4-fde292f1a18a","Type":"ContainerDied","Data":"e9047893659a8e2b790226f952cce36dfc29c02bd00a0e2394185df2cca590eb"} Sep 19 08:32:16.954863 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.954597 2585 scope.go:117] "RemoveContainer" containerID="d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919" Sep 19 08:32:16.955079 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.954982 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" Sep 19 08:32:16.963291 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.963276 2585 scope.go:117] "RemoveContainer" containerID="0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f" Sep 19 08:32:16.970409 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.970392 2585 scope.go:117] "RemoveContainer" containerID="d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919" Sep 19 08:32:16.970697 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:16.970676 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919\": container with ID starting with d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919 not found: ID does not exist" containerID="d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919" Sep 19 08:32:16.970768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.970706 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919"} err="failed to get container status \"d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919\": rpc error: code = NotFound desc = could not find container \"d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919\": container with ID starting with d885de1e6a12d8a75dca14dc6405c46cdfd2b9c389cad0782efca2316c152919 not found: ID does not exist" Sep 19 08:32:16.970768 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.970735 2585 scope.go:117] "RemoveContainer" containerID="0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f" Sep 19 08:32:16.970998 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:16.970980 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f\": container with ID starting with 0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f not found: ID does not exist" containerID="0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f" Sep 19 08:32:16.971049 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.971004 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f"} err="failed to get container status \"0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f\": rpc error: code = NotFound desc = could not find container \"0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f\": container with ID starting with 0d5a359665aa1adeb39a424f2b89346f9824d1d734f546d2066519a904bebb3f not found: ID does not exist" Sep 19 08:32:16.975431 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.975410 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j"] Sep 19 08:32:16.979153 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:16.979133 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-83f4f-predictor-749b7d56dc-l9k9j"] Sep 19 08:32:17.305896 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:17.305818 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" path="/var/lib/kubelet/pods/f0b36ea0-f156-4b94-9be4-fde292f1a18a/volumes" Sep 19 08:32:20.906002 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:20.905965 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:32:21.960105 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:21.960072 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:32:21.960567 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:21.960515 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" Sep 19 08:32:25.906136 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:25.906049 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:32:25.906513 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:25.906166 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:30.906266 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:30.906221 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:32:31.960500 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:31.960465 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" Sep 19 08:32:35.906281 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:35.906235 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:32:40.905982 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:40.905944 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:32:41.961088 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:41.961049 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" Sep 19 08:32:42.628911 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:42.628880 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:42.727000 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:42.726916 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls\") pod \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " Sep 19 08:32:42.727136 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:42.727003 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-openshift-service-ca-bundle\") pod \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\" (UID: \"aa193bdf-6ea6-4cb7-a374-b7db1d496f17\") " Sep 19 08:32:42.727345 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:42.727321 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "aa193bdf-6ea6-4cb7-a374-b7db1d496f17" (UID: "aa193bdf-6ea6-4cb7-a374-b7db1d496f17"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:32:42.728941 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:42.728916 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aa193bdf-6ea6-4cb7-a374-b7db1d496f17" (UID: "aa193bdf-6ea6-4cb7-a374-b7db1d496f17"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:32:42.827444 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:42.827410 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:32:42.827444 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:42.827437 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aa193bdf-6ea6-4cb7-a374-b7db1d496f17-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:32:43.053315 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.053231 2585 generic.go:356] "Generic (PLEG): container finished" podID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerID="c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d" exitCode=0 Sep 19 08:32:43.053315 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.053279 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" Sep 19 08:32:43.053797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.053314 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" event={"ID":"aa193bdf-6ea6-4cb7-a374-b7db1d496f17","Type":"ContainerDied","Data":"c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d"} Sep 19 08:32:43.053797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.053365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb" event={"ID":"aa193bdf-6ea6-4cb7-a374-b7db1d496f17","Type":"ContainerDied","Data":"8e934a049e12f933dbd9a605ff83465319df9421e06579a6cefea650bc4d1e01"} Sep 19 08:32:43.053797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.053388 2585 scope.go:117] "RemoveContainer" containerID="c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d" Sep 19 08:32:43.063487 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.063467 2585 scope.go:117] "RemoveContainer" containerID="c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d" Sep 19 08:32:43.063791 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:43.063771 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d\": container with ID starting with c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d not found: ID does not exist" containerID="c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d" Sep 19 08:32:43.063844 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.063801 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d"} err="failed to get container status \"c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d\": rpc error: code = NotFound desc = could not find container \"c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d\": container with ID starting with c9e0048dea103b4eac05c7955831a9ff3608f7201a832bcd874642b27df22e1d not found: ID does not exist" Sep 19 08:32:43.075026 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.075003 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb"] Sep 19 08:32:43.078775 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.078748 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-83f4f-564cdf5744-btvcb"] Sep 19 08:32:43.305614 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:43.305537 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" path="/var/lib/kubelet/pods/aa193bdf-6ea6-4cb7-a374-b7db1d496f17/volumes" Sep 19 08:32:51.437217 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437183 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9"] Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437472 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kube-rbac-proxy" Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437483 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kube-rbac-proxy" Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437495 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437501 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437515 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437532 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437579 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kube-rbac-proxy" Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437586 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0b36ea0-f156-4b94-9be4-fde292f1a18a" containerName="kserve-container" Sep 19 08:32:51.437667 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.437593 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa193bdf-6ea6-4cb7-a374-b7db1d496f17" containerName="switch-graph-83f4f" Sep 19 08:32:51.441836 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.441817 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:51.444282 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.444261 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-kube-rbac-proxy-sar-config\"" Sep 19 08:32:51.444388 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.444261 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"model-chainer-serving-cert\"" Sep 19 08:32:51.448753 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.448470 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9"] Sep 19 08:32:51.492040 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.492007 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b689d9-774f-4b08-acb3-018ed8fd5c28-openshift-service-ca-bundle\") pod \"model-chainer-7bcc66cdd8-jgzm9\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:51.492207 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.492075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls\") pod \"model-chainer-7bcc66cdd8-jgzm9\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:51.593307 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.593274 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls\") pod \"model-chainer-7bcc66cdd8-jgzm9\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:51.593472 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.593341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b689d9-774f-4b08-acb3-018ed8fd5c28-openshift-service-ca-bundle\") pod \"model-chainer-7bcc66cdd8-jgzm9\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:51.593472 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:51.593451 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Sep 19 08:32:51.593568 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:51.593539 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls podName:05b689d9-774f-4b08-acb3-018ed8fd5c28 nodeName:}" failed. No retries permitted until 2026-09-19 08:32:52.093505448 +0000 UTC m=+741.305339290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls") pod "model-chainer-7bcc66cdd8-jgzm9" (UID: "05b689d9-774f-4b08-acb3-018ed8fd5c28") : secret "model-chainer-serving-cert" not found Sep 19 08:32:51.594008 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.593987 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b689d9-774f-4b08-acb3-018ed8fd5c28-openshift-service-ca-bundle\") pod \"model-chainer-7bcc66cdd8-jgzm9\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:51.960607 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:51.960574 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" Sep 19 08:32:52.097578 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:52.097540 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls\") pod \"model-chainer-7bcc66cdd8-jgzm9\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:52.097739 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:52.097708 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/model-chainer-serving-cert: secret "model-chainer-serving-cert" not found Sep 19 08:32:52.097813 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:32:52.097803 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls podName:05b689d9-774f-4b08-acb3-018ed8fd5c28 nodeName:}" failed. No retries permitted until 2026-09-19 08:32:53.09778874 +0000 UTC m=+742.309622579 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls") pod "model-chainer-7bcc66cdd8-jgzm9" (UID: "05b689d9-774f-4b08-acb3-018ed8fd5c28") : secret "model-chainer-serving-cert" not found Sep 19 08:32:53.105537 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:53.105476 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls\") pod \"model-chainer-7bcc66cdd8-jgzm9\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:53.107898 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:53.107877 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls\") pod \"model-chainer-7bcc66cdd8-jgzm9\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:53.252564 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:53.252508 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:53.521372 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:53.521286 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9"] Sep 19 08:32:53.524135 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:32:53.524107 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b689d9_774f_4b08_acb3_018ed8fd5c28.slice/crio-d87ce66b3e9ad0c6b0ec4c2ce8065ffad724b6a8856fafe23dcf069b39abfc2a WatchSource:0}: Error finding container d87ce66b3e9ad0c6b0ec4c2ce8065ffad724b6a8856fafe23dcf069b39abfc2a: Status 404 returned error can't find the container with id d87ce66b3e9ad0c6b0ec4c2ce8065ffad724b6a8856fafe23dcf069b39abfc2a Sep 19 08:32:54.096407 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:54.096356 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" event={"ID":"05b689d9-774f-4b08-acb3-018ed8fd5c28","Type":"ContainerStarted","Data":"c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6"} Sep 19 08:32:54.096596 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:54.096424 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" event={"ID":"05b689d9-774f-4b08-acb3-018ed8fd5c28","Type":"ContainerStarted","Data":"d87ce66b3e9ad0c6b0ec4c2ce8065ffad724b6a8856fafe23dcf069b39abfc2a"} Sep 19 08:32:54.096596 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:54.096452 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:32:54.113105 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:32:54.113058 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" podStartSLOduration=3.113042191 podStartE2EDuration="3.113042191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:32:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:32:54.111620645 +0000 UTC m=+743.323454501" watchObservedRunningTime="2026-09-19 08:32:54.113042191 +0000 UTC m=+743.324876048" Sep 19 08:33:00.105978 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:00.105950 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:33:01.961571 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:01.961532 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.21:8080: connect: connection refused" Sep 19 08:33:02.980231 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:02.980201 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9"] Sep 19 08:33:02.980670 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:02.980365 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" containerID="cri-o://c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6" gracePeriod=30 Sep 19 08:33:03.306295 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.306221 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d"] Sep 19 08:33:03.309490 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.309474 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.311829 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.311798 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5cf0a-predictor-serving-cert\"" Sep 19 08:33:03.311933 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.311798 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5cf0a-kube-rbac-proxy-sar-config\"" Sep 19 08:33:03.318684 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.318663 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d"] Sep 19 08:33:03.381343 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.381306 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.381549 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.381356 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nnnh\" (UniqueName: \"kubernetes.io/projected/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-api-access-5nnnh\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.381549 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.381477 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f64b97-8ae5-4690-a718-01a24bc5062f-proxy-tls\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.482788 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.482745 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f64b97-8ae5-4690-a718-01a24bc5062f-proxy-tls\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.482985 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.482805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.482985 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.482860 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nnnh\" (UniqueName: \"kubernetes.io/projected/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-api-access-5nnnh\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.483557 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.483507 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.485185 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.485164 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f64b97-8ae5-4690-a718-01a24bc5062f-proxy-tls\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.491813 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.491786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nnnh\" (UniqueName: \"kubernetes.io/projected/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-api-access-5nnnh\") pod \"success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.619119 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.619037 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:03.889064 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:03.888977 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d"] Sep 19 08:33:03.891946 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:33:03.891917 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f64b97_8ae5_4690_a718_01a24bc5062f.slice/crio-0e5020cd0ec05badedbca2499d68e610d030d14677255788a623295321c9d57a WatchSource:0}: Error finding container 0e5020cd0ec05badedbca2499d68e610d030d14677255788a623295321c9d57a: Status 404 returned error can't find the container with id 0e5020cd0ec05badedbca2499d68e610d030d14677255788a623295321c9d57a Sep 19 08:33:04.141084 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:04.141052 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" event={"ID":"c6f64b97-8ae5-4690-a718-01a24bc5062f","Type":"ContainerStarted","Data":"e171cc98831c716cbf37fa8c7ae031b3073835eaa77264a4e2a867927f7c1f0c"} Sep 19 08:33:04.141465 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:04.141108 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" event={"ID":"c6f64b97-8ae5-4690-a718-01a24bc5062f","Type":"ContainerStarted","Data":"94375871756e902fa1507c74cae475f0f6c2640c2d1e756afbd52418842bdb55"} Sep 19 08:33:04.141465 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:04.141134 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" event={"ID":"c6f64b97-8ae5-4690-a718-01a24bc5062f","Type":"ContainerStarted","Data":"0e5020cd0ec05badedbca2499d68e610d030d14677255788a623295321c9d57a"} Sep 19 08:33:04.141465 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:04.141243 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:04.141465 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:04.141302 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:04.142585 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:04.142558 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Sep 19 08:33:04.158968 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:04.158920 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podStartSLOduration=1.158906933 podStartE2EDuration="1.158906933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:33:04.156813724 +0000 UTC m=+753.368647581" watchObservedRunningTime="2026-09-19 08:33:04.158906933 +0000 UTC m=+753.370740870" Sep 19 08:33:05.105101 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:05.105056 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:33:05.150698 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:05.148850 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Sep 19 08:33:10.104978 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:10.104944 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:33:10.153496 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:10.153468 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:33:10.154008 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:10.153984 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Sep 19 08:33:11.961515 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:11.961478 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:33:15.104212 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:15.104173 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:33:15.104614 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:15.104273 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:33:20.104699 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:20.104653 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:33:20.154204 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:20.154166 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Sep 19 08:33:22.684608 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.684578 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v"] Sep 19 08:33:22.689069 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.689052 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:22.691529 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.691494 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-96102-serving-cert\"" Sep 19 08:33:22.691655 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.691634 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-96102-kube-rbac-proxy-sar-config\"" Sep 19 08:33:22.694598 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.694572 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v"] Sep 19 08:33:22.841708 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.841675 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-openshift-service-ca-bundle\") pod \"switch-graph-96102-85669f7948-vb92v\" (UID: \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\") " pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:22.841708 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.841704 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-proxy-tls\") pod \"switch-graph-96102-85669f7948-vb92v\" (UID: \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\") " pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:22.942562 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.942461 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-openshift-service-ca-bundle\") pod \"switch-graph-96102-85669f7948-vb92v\" (UID: \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\") " pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:22.942562 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.942495 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-proxy-tls\") pod \"switch-graph-96102-85669f7948-vb92v\" (UID: \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\") " pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:22.943157 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.943131 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-openshift-service-ca-bundle\") pod \"switch-graph-96102-85669f7948-vb92v\" (UID: \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\") " pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:22.944937 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:22.944918 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-proxy-tls\") pod \"switch-graph-96102-85669f7948-vb92v\" (UID: \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\") " pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:23.000187 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:23.000161 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:23.267835 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:23.267751 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v"] Sep 19 08:33:23.270552 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:33:23.270506 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6468d9_cdb2_4b72_9684_14b9e14ece9e.slice/crio-902d2aeffe1cd8ac3f11c1c59240f71fe5a40b41b87ac48f8d54f2e648958d62 WatchSource:0}: Error finding container 902d2aeffe1cd8ac3f11c1c59240f71fe5a40b41b87ac48f8d54f2e648958d62: Status 404 returned error can't find the container with id 902d2aeffe1cd8ac3f11c1c59240f71fe5a40b41b87ac48f8d54f2e648958d62 Sep 19 08:33:24.227877 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:24.227837 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" event={"ID":"bc6468d9-cdb2-4b72-9684-14b9e14ece9e","Type":"ContainerStarted","Data":"d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a"} Sep 19 08:33:24.228260 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:24.227895 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" event={"ID":"bc6468d9-cdb2-4b72-9684-14b9e14ece9e","Type":"ContainerStarted","Data":"902d2aeffe1cd8ac3f11c1c59240f71fe5a40b41b87ac48f8d54f2e648958d62"} Sep 19 08:33:24.228260 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:24.227956 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:24.242953 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:24.242905 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" podStartSLOduration=2.242888109 podStartE2EDuration="2.242888109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:33:24.242311626 +0000 UTC m=+773.454145484" watchObservedRunningTime="2026-09-19 08:33:24.242888109 +0000 UTC m=+773.454721967" Sep 19 08:33:25.105066 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:25.105026 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:33:30.104985 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:30.104944 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:33:30.154090 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:30.154049 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Sep 19 08:33:30.238792 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:30.238764 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:33:33.123122 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.123093 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:33:33.233431 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.233388 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b689d9-774f-4b08-acb3-018ed8fd5c28-openshift-service-ca-bundle\") pod \"05b689d9-774f-4b08-acb3-018ed8fd5c28\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " Sep 19 08:33:33.233431 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.233440 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls\") pod \"05b689d9-774f-4b08-acb3-018ed8fd5c28\" (UID: \"05b689d9-774f-4b08-acb3-018ed8fd5c28\") " Sep 19 08:33:33.233787 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.233762 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b689d9-774f-4b08-acb3-018ed8fd5c28-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "05b689d9-774f-4b08-acb3-018ed8fd5c28" (UID: "05b689d9-774f-4b08-acb3-018ed8fd5c28"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:33:33.235441 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.235385 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05b689d9-774f-4b08-acb3-018ed8fd5c28" (UID: "05b689d9-774f-4b08-acb3-018ed8fd5c28"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:33:33.262035 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.262004 2585 generic.go:356] "Generic (PLEG): container finished" podID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerID="c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6" exitCode=0 Sep 19 08:33:33.262150 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.262045 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" Sep 19 08:33:33.262150 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.262060 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" event={"ID":"05b689d9-774f-4b08-acb3-018ed8fd5c28","Type":"ContainerDied","Data":"c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6"} Sep 19 08:33:33.262150 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.262102 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9" event={"ID":"05b689d9-774f-4b08-acb3-018ed8fd5c28","Type":"ContainerDied","Data":"d87ce66b3e9ad0c6b0ec4c2ce8065ffad724b6a8856fafe23dcf069b39abfc2a"} Sep 19 08:33:33.262150 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.262125 2585 scope.go:117] "RemoveContainer" containerID="c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6" Sep 19 08:33:33.270053 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.270033 2585 scope.go:117] "RemoveContainer" containerID="c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6" Sep 19 08:33:33.270292 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:33:33.270276 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6\": container with ID starting with c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6 not found: ID does not exist" containerID="c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6" Sep 19 08:33:33.270330 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.270301 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6"} err="failed to get container status \"c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6\": rpc error: code = NotFound desc = could not find container \"c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6\": container with ID starting with c51228f19360beb1405365095e64a8f59191a35f9be238f0c64a7cc86771b5b6 not found: ID does not exist" Sep 19 08:33:33.280603 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.280579 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9"] Sep 19 08:33:33.285307 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.285285 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/model-chainer-7bcc66cdd8-jgzm9"] Sep 19 08:33:33.305039 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.305014 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" path="/var/lib/kubelet/pods/05b689d9-774f-4b08-acb3-018ed8fd5c28/volumes" Sep 19 08:33:33.334458 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.334440 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05b689d9-774f-4b08-acb3-018ed8fd5c28-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:33:33.334458 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:33.334457 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05b689d9-774f-4b08-acb3-018ed8fd5c28-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:33:40.154678 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:40.154639 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Sep 19 08:33:50.154149 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:33:50.154100 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Sep 19 08:34:00.155580 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:00.155488 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:34:13.179787 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.179740 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z"] Sep 19 08:34:13.180269 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.180179 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" Sep 19 08:34:13.180269 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.180200 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" Sep 19 08:34:13.180378 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.180281 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="05b689d9-774f-4b08-acb3-018ed8fd5c28" containerName="model-chainer" Sep 19 08:34:13.183300 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.183271 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:13.185727 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.185704 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-5cf0a-kube-rbac-proxy-sar-config\"" Sep 19 08:34:13.185920 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.185789 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-5cf0a-serving-cert\"" Sep 19 08:34:13.192357 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.192331 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z"] Sep 19 08:34:13.255833 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.255789 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fbaa75-267c-4473-bd2e-df43c40ca70c-openshift-service-ca-bundle\") pod \"sequence-graph-5cf0a-5bd5db7dc4-x978z\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:13.255833 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.255834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls\") pod \"sequence-graph-5cf0a-5bd5db7dc4-x978z\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:13.356299 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.356255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fbaa75-267c-4473-bd2e-df43c40ca70c-openshift-service-ca-bundle\") pod \"sequence-graph-5cf0a-5bd5db7dc4-x978z\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:13.356478 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.356309 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls\") pod \"sequence-graph-5cf0a-5bd5db7dc4-x978z\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:13.356478 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:34:13.356454 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-5cf0a-serving-cert: secret "sequence-graph-5cf0a-serving-cert" not found Sep 19 08:34:13.356602 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:34:13.356517 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls podName:95fbaa75-267c-4473-bd2e-df43c40ca70c nodeName:}" failed. No retries permitted until 2026-09-19 08:34:13.856501354 +0000 UTC m=+823.068335189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls") pod "sequence-graph-5cf0a-5bd5db7dc4-x978z" (UID: "95fbaa75-267c-4473-bd2e-df43c40ca70c") : secret "sequence-graph-5cf0a-serving-cert" not found Sep 19 08:34:13.356951 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.356928 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fbaa75-267c-4473-bd2e-df43c40ca70c-openshift-service-ca-bundle\") pod \"sequence-graph-5cf0a-5bd5db7dc4-x978z\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:13.859727 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.859693 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls\") pod \"sequence-graph-5cf0a-5bd5db7dc4-x978z\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:13.862209 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:13.862179 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls\") pod \"sequence-graph-5cf0a-5bd5db7dc4-x978z\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:14.095953 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:14.095909 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:14.364501 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:14.364432 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z"] Sep 19 08:34:14.367164 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:34:14.367137 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95fbaa75_267c_4473_bd2e_df43c40ca70c.slice/crio-d5b2d9d08fc263588d8280916522b17f2b0329d4ff18c64efab4a467760033d2 WatchSource:0}: Error finding container d5b2d9d08fc263588d8280916522b17f2b0329d4ff18c64efab4a467760033d2: Status 404 returned error can't find the container with id d5b2d9d08fc263588d8280916522b17f2b0329d4ff18c64efab4a467760033d2 Sep 19 08:34:14.415043 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:14.415015 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" event={"ID":"95fbaa75-267c-4473-bd2e-df43c40ca70c","Type":"ContainerStarted","Data":"d5b2d9d08fc263588d8280916522b17f2b0329d4ff18c64efab4a467760033d2"} Sep 19 08:34:15.419658 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:15.419626 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" event={"ID":"95fbaa75-267c-4473-bd2e-df43c40ca70c","Type":"ContainerStarted","Data":"06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96"} Sep 19 08:34:15.420089 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:15.419690 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:34:15.435788 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:15.435739 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" podStartSLOduration=2.435722756 podStartE2EDuration="2.435722756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:34:15.434576708 +0000 UTC m=+824.646410564" watchObservedRunningTime="2026-09-19 08:34:15.435722756 +0000 UTC m=+824.647556612" Sep 19 08:34:21.428999 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:34:21.428970 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:35:31.206576 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:35:31.206476 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:35:31.207891 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:35:31.207864 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:40:31.230119 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:40:31.230029 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:40:31.232710 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:40:31.232690 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:41:37.898117 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:37.898041 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v"] Sep 19 08:41:37.900562 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:37.898251 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" containerID="cri-o://d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a" gracePeriod=30 Sep 19 08:41:38.019564 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.019534 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6"] Sep 19 08:41:38.019789 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.019760 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" containerID="cri-o://97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404" gracePeriod=30 Sep 19 08:41:38.019861 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.019807 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kube-rbac-proxy" containerID="cri-o://60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495" gracePeriod=30 Sep 19 08:41:38.118413 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.118379 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n"] Sep 19 08:41:38.122243 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.122226 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.124952 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.124925 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-88665-predictor-serving-cert\"" Sep 19 08:41:38.125064 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.124979 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-88665-kube-rbac-proxy-sar-config\"" Sep 19 08:41:38.130394 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.130297 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n"] Sep 19 08:41:38.233881 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.233801 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.233881 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.233836 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w48tg\" (UniqueName: \"kubernetes.io/projected/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-api-access-w48tg\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.233881 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.233857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.334961 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.334928 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.334961 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.334961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w48tg\" (UniqueName: \"kubernetes.io/projected/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-api-access-w48tg\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.335177 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.334994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.335177 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:41:38.335138 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-88665-predictor-serving-cert: secret "success-200-isvc-88665-predictor-serving-cert" not found Sep 19 08:41:38.335256 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:41:38.335197 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls podName:497df55c-e6d1-4283-a4c7-c0ec9db45a72 nodeName:}" failed. No retries permitted until 2026-09-19 08:41:38.835176996 +0000 UTC m=+1268.047010846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls") pod "success-200-isvc-88665-predictor-585866c4c-t4g2n" (UID: "497df55c-e6d1-4283-a4c7-c0ec9db45a72") : secret "success-200-isvc-88665-predictor-serving-cert" not found Sep 19 08:41:38.335721 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.335702 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.343339 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.343316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w48tg\" (UniqueName: \"kubernetes.io/projected/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-api-access-w48tg\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.838214 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:38.838181 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:38.838411 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:41:38.838297 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-88665-predictor-serving-cert: secret "success-200-isvc-88665-predictor-serving-cert" not found Sep 19 08:41:38.838411 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:41:38.838347 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls podName:497df55c-e6d1-4283-a4c7-c0ec9db45a72 nodeName:}" failed. No retries permitted until 2026-09-19 08:41:39.838335067 +0000 UTC m=+1269.050168901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls") pod "success-200-isvc-88665-predictor-585866c4c-t4g2n" (UID: "497df55c-e6d1-4283-a4c7-c0ec9db45a72") : secret "success-200-isvc-88665-predictor-serving-cert" not found Sep 19 08:41:39.090551 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:39.090444 2585 generic.go:356] "Generic (PLEG): container finished" podID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerID="60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495" exitCode=2 Sep 19 08:41:39.090551 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:39.090514 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" event={"ID":"ee89b915-2e78-4f8d-8773-770ba9eb6d4c","Type":"ContainerDied","Data":"60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495"} Sep 19 08:41:39.846210 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:39.846168 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:39.848658 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:39.848632 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls\") pod \"success-200-isvc-88665-predictor-585866c4c-t4g2n\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:39.934367 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:39.934340 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:40.204384 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:40.204340 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n"] Sep 19 08:41:40.206706 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:41:40.206678 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497df55c_e6d1_4283_a4c7_c0ec9db45a72.slice/crio-634f8798d73a11a1be786133c56765e7b8164f7db38df86a581328f2e49fbb56 WatchSource:0}: Error finding container 634f8798d73a11a1be786133c56765e7b8164f7db38df86a581328f2e49fbb56: Status 404 returned error can't find the container with id 634f8798d73a11a1be786133c56765e7b8164f7db38df86a581328f2e49fbb56 Sep 19 08:41:40.237006 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:40.236975 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:41:41.099743 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.099707 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" event={"ID":"497df55c-e6d1-4283-a4c7-c0ec9db45a72","Type":"ContainerStarted","Data":"c4f79e7aae0ba5e6a281c7c4eee4bc966386db0ca70aa8b0c224bb2447284cc0"} Sep 19 08:41:41.099887 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.099767 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" event={"ID":"497df55c-e6d1-4283-a4c7-c0ec9db45a72","Type":"ContainerStarted","Data":"599d32d3ad057ddd38a981e1d25ebee299f4f4111854f877782a5abe1db4f3aa"} Sep 19 08:41:41.099887 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.099787 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" event={"ID":"497df55c-e6d1-4283-a4c7-c0ec9db45a72","Type":"ContainerStarted","Data":"634f8798d73a11a1be786133c56765e7b8164f7db38df86a581328f2e49fbb56"} Sep 19 08:41:41.099887 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.099853 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:41.117079 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.117033 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podStartSLOduration=3.117019507 podStartE2EDuration="3.117019507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:41:41.115877979 +0000 UTC m=+1270.327711836" watchObservedRunningTime="2026-09-19 08:41:41.117019507 +0000 UTC m=+1270.328853364" Sep 19 08:41:41.367279 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.367255 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:41:41.461288 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.461258 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-rbac-proxy-sar-config\") pod \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " Sep 19 08:41:41.461439 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.461337 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls\") pod \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " Sep 19 08:41:41.461439 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.461358 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9r8hk\" (UniqueName: \"kubernetes.io/projected/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-api-access-9r8hk\") pod \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\" (UID: \"ee89b915-2e78-4f8d-8773-770ba9eb6d4c\") " Sep 19 08:41:41.461718 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.461691 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ee89b915-2e78-4f8d-8773-770ba9eb6d4c" (UID: "ee89b915-2e78-4f8d-8773-770ba9eb6d4c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:41:41.463420 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.463393 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ee89b915-2e78-4f8d-8773-770ba9eb6d4c" (UID: "ee89b915-2e78-4f8d-8773-770ba9eb6d4c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:41:41.463538 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.463421 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-api-access-9r8hk" (OuterVolumeSpecName: "kube-api-access-9r8hk") pod "ee89b915-2e78-4f8d-8773-770ba9eb6d4c" (UID: "ee89b915-2e78-4f8d-8773-770ba9eb6d4c"). InnerVolumeSpecName "kube-api-access-9r8hk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 08:41:41.562662 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.562629 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:41:41.562662 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.562655 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:41:41.562662 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:41.562664 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r8hk\" (UniqueName: \"kubernetes.io/projected/ee89b915-2e78-4f8d-8773-770ba9eb6d4c-kube-api-access-9r8hk\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:41:42.105235 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.105202 2585 generic.go:356] "Generic (PLEG): container finished" podID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerID="97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404" exitCode=0 Sep 19 08:41:42.105432 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.105244 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" Sep 19 08:41:42.105432 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.105285 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" event={"ID":"ee89b915-2e78-4f8d-8773-770ba9eb6d4c","Type":"ContainerDied","Data":"97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404"} Sep 19 08:41:42.105432 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.105358 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6" event={"ID":"ee89b915-2e78-4f8d-8773-770ba9eb6d4c","Type":"ContainerDied","Data":"39191feea04e10bceacc0f72b19fea426998b22491ff4e6a04655fdfb21e8d7e"} Sep 19 08:41:42.105432 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.105399 2585 scope.go:117] "RemoveContainer" containerID="60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495" Sep 19 08:41:42.105851 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.105827 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:42.107321 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.107295 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 19 08:41:42.113570 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.113549 2585 scope.go:117] "RemoveContainer" containerID="97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404" Sep 19 08:41:42.120710 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.120694 2585 scope.go:117] "RemoveContainer" containerID="60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495" Sep 19 08:41:42.120973 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:41:42.120955 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495\": container with ID starting with 60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495 not found: ID does not exist" containerID="60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495" Sep 19 08:41:42.121046 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.120984 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495"} err="failed to get container status \"60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495\": rpc error: code = NotFound desc = could not find container \"60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495\": container with ID starting with 60e7df8f262ce4e484844890d3fb9fa169c51016fdefeca839f04b3f700d6495 not found: ID does not exist" Sep 19 08:41:42.121046 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.121006 2585 scope.go:117] "RemoveContainer" containerID="97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404" Sep 19 08:41:42.121234 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:41:42.121218 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404\": container with ID starting with 97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404 not found: ID does not exist" containerID="97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404" Sep 19 08:41:42.121285 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.121242 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404"} err="failed to get container status \"97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404\": rpc error: code = NotFound desc = could not find container \"97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404\": container with ID starting with 97e44e4682751101d8e5467e097d1cd51df216b45d1ab4a07365afd76a8d8404 not found: ID does not exist" Sep 19 08:41:42.126428 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.126408 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6"] Sep 19 08:41:42.129700 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:42.129680 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-96102-predictor-6db489744f-tmcb6"] Sep 19 08:41:43.110290 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:43.110254 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 19 08:41:43.306025 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:43.305992 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" path="/var/lib/kubelet/pods/ee89b915-2e78-4f8d-8773-770ba9eb6d4c/volumes" Sep 19 08:41:45.236452 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:45.236418 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:41:48.114808 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:48.114776 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:41:48.115238 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:48.115167 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 19 08:41:50.236120 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:50.236078 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:41:50.236564 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:50.236227 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:41:55.237386 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:55.237342 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:41:58.115440 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:41:58.115405 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 19 08:42:00.237043 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:00.237002 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:42:05.236538 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:05.236488 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:42:08.051273 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.051249 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:42:08.115550 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.115478 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 19 08:42:08.181615 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.181515 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-proxy-tls\") pod \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\" (UID: \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\") " Sep 19 08:42:08.181773 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.181626 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-openshift-service-ca-bundle\") pod \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\" (UID: \"bc6468d9-cdb2-4b72-9684-14b9e14ece9e\") " Sep 19 08:42:08.181975 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.181952 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "bc6468d9-cdb2-4b72-9684-14b9e14ece9e" (UID: "bc6468d9-cdb2-4b72-9684-14b9e14ece9e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:42:08.183602 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.183572 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bc6468d9-cdb2-4b72-9684-14b9e14ece9e" (UID: "bc6468d9-cdb2-4b72-9684-14b9e14ece9e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:42:08.204029 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.203996 2585 generic.go:356] "Generic (PLEG): container finished" podID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerID="d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a" exitCode=0 Sep 19 08:42:08.204125 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.204039 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" Sep 19 08:42:08.204125 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.204083 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" event={"ID":"bc6468d9-cdb2-4b72-9684-14b9e14ece9e","Type":"ContainerDied","Data":"d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a"} Sep 19 08:42:08.204195 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.204164 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v" event={"ID":"bc6468d9-cdb2-4b72-9684-14b9e14ece9e","Type":"ContainerDied","Data":"902d2aeffe1cd8ac3f11c1c59240f71fe5a40b41b87ac48f8d54f2e648958d62"} Sep 19 08:42:08.204229 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.204193 2585 scope.go:117] "RemoveContainer" containerID="d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a" Sep 19 08:42:08.212568 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.212548 2585 scope.go:117] "RemoveContainer" containerID="d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a" Sep 19 08:42:08.212844 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:42:08.212825 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a\": container with ID starting with d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a not found: ID does not exist" containerID="d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a" Sep 19 08:42:08.212898 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.212857 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a"} err="failed to get container status \"d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a\": rpc error: code = NotFound desc = could not find container \"d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a\": container with ID starting with d90a11d2705694833eedeeab224d63636aa38ee49f5f618b3f6599b7407fdb0a not found: ID does not exist" Sep 19 08:42:08.225028 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.225007 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v"] Sep 19 08:42:08.228839 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.228820 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-96102-85669f7948-vb92v"] Sep 19 08:42:08.282251 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.282233 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:42:08.282251 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:08.282251 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc6468d9-cdb2-4b72-9684-14b9e14ece9e-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:42:09.305553 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:09.305502 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" path="/var/lib/kubelet/pods/bc6468d9-cdb2-4b72-9684-14b9e14ece9e/volumes" Sep 19 08:42:18.115821 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:18.115783 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 19 08:42:28.115202 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.115158 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused" Sep 19 08:42:28.735800 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.735767 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z"] Sep 19 08:42:28.735995 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.735972 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" containerID="cri-o://06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96" gracePeriod=30 Sep 19 08:42:28.839249 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.839220 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d"] Sep 19 08:42:28.839468 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.839438 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" containerID="cri-o://94375871756e902fa1507c74cae475f0f6c2640c2d1e756afbd52418842bdb55" gracePeriod=30 Sep 19 08:42:28.839641 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.839482 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kube-rbac-proxy" containerID="cri-o://e171cc98831c716cbf37fa8c7ae031b3073835eaa77264a4e2a867927f7c1f0c" gracePeriod=30 Sep 19 08:42:28.979770 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.979737 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg"] Sep 19 08:42:28.980200 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980179 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" Sep 19 08:42:28.980243 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980206 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" Sep 19 08:42:28.980243 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980237 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kube-rbac-proxy" Sep 19 08:42:28.980309 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980245 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kube-rbac-proxy" Sep 19 08:42:28.980309 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980255 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" Sep 19 08:42:28.980309 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980264 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" Sep 19 08:42:28.980390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980343 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc6468d9-cdb2-4b72-9684-14b9e14ece9e" containerName="switch-graph-96102" Sep 19 08:42:28.980390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980357 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kserve-container" Sep 19 08:42:28.980390 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.980366 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee89b915-2e78-4f8d-8773-770ba9eb6d4c" containerName="kube-rbac-proxy" Sep 19 08:42:28.983621 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.983604 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:28.985988 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.985928 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3bcba-kube-rbac-proxy-sar-config\"" Sep 19 08:42:28.986111 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.986076 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3bcba-predictor-serving-cert\"" Sep 19 08:42:28.993384 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:28.993365 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg"] Sep 19 08:42:29.045275 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.045236 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.045419 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.045294 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655jt\" (UniqueName: \"kubernetes.io/projected/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-api-access-655jt\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.045419 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.045349 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.145846 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.145810 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.146213 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.145899 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.146213 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.145923 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655jt\" (UniqueName: \"kubernetes.io/projected/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-api-access-655jt\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.146213 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:42:29.146041 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-serving-cert: secret "success-200-isvc-3bcba-predictor-serving-cert" not found Sep 19 08:42:29.146213 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:42:29.146107 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls podName:03d4b4f1-60e6-4d89-a40b-4b41bc7b731e nodeName:}" failed. No retries permitted until 2026-09-19 08:42:29.646091335 +0000 UTC m=+1318.857925170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls") pod "success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" (UID: "03d4b4f1-60e6-4d89-a40b-4b41bc7b731e") : secret "success-200-isvc-3bcba-predictor-serving-cert" not found Sep 19 08:42:29.146479 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.146459 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.156800 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.156775 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-655jt\" (UniqueName: \"kubernetes.io/projected/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-api-access-655jt\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.289456 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.289356 2585 generic.go:356] "Generic (PLEG): container finished" podID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerID="e171cc98831c716cbf37fa8c7ae031b3073835eaa77264a4e2a867927f7c1f0c" exitCode=2 Sep 19 08:42:29.289456 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.289431 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" event={"ID":"c6f64b97-8ae5-4690-a718-01a24bc5062f","Type":"ContainerDied","Data":"e171cc98831c716cbf37fa8c7ae031b3073835eaa77264a4e2a867927f7c1f0c"} Sep 19 08:42:29.648653 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.648617 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.650970 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.650939 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls\") pod \"success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:29.894096 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:29.894055 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:30.149246 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:30.149207 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused" Sep 19 08:42:30.154112 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:30.154039 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused" Sep 19 08:42:30.163700 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:30.163678 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg"] Sep 19 08:42:30.166370 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:42:30.166344 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03d4b4f1_60e6_4d89_a40b_4b41bc7b731e.slice/crio-ebef8b0dc35fc71cfdfddf4a0a78e9edd90fa54e43fdf895b1d03b01531d9683 WatchSource:0}: Error finding container ebef8b0dc35fc71cfdfddf4a0a78e9edd90fa54e43fdf895b1d03b01531d9683: Status 404 returned error can't find the container with id ebef8b0dc35fc71cfdfddf4a0a78e9edd90fa54e43fdf895b1d03b01531d9683 Sep 19 08:42:30.296329 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:30.296299 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" event={"ID":"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e","Type":"ContainerStarted","Data":"6edf64d816b9b9c3ef910401d835b8be4fa402009f5bd915092efc8df33d2ddc"} Sep 19 08:42:30.296474 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:30.296355 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" event={"ID":"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e","Type":"ContainerStarted","Data":"555a01d8b994af46fae839c380c9ad5085e05beb1473c8fed038af39990ac5ad"} Sep 19 08:42:30.296474 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:30.296374 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" event={"ID":"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e","Type":"ContainerStarted","Data":"ebef8b0dc35fc71cfdfddf4a0a78e9edd90fa54e43fdf895b1d03b01531d9683"} Sep 19 08:42:30.296612 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:30.296515 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:30.315304 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:30.315261 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podStartSLOduration=2.3152459309999998 podStartE2EDuration="2.315245931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:42:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:42:30.314642262 +0000 UTC m=+1319.526476121" watchObservedRunningTime="2026-09-19 08:42:30.315245931 +0000 UTC m=+1319.527079789" Sep 19 08:42:31.300090 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:31.300060 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:31.301482 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:31.301454 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 19 08:42:31.426966 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:31.426927 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:42:32.307131 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.307098 2585 generic.go:356] "Generic (PLEG): container finished" podID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerID="94375871756e902fa1507c74cae475f0f6c2640c2d1e756afbd52418842bdb55" exitCode=0 Sep 19 08:42:32.307550 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.307170 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" event={"ID":"c6f64b97-8ae5-4690-a718-01a24bc5062f","Type":"ContainerDied","Data":"94375871756e902fa1507c74cae475f0f6c2640c2d1e756afbd52418842bdb55"} Sep 19 08:42:32.307673 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.307644 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 19 08:42:32.388021 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.387995 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:42:32.467769 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.467689 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5nnnh\" (UniqueName: \"kubernetes.io/projected/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-api-access-5nnnh\") pod \"c6f64b97-8ae5-4690-a718-01a24bc5062f\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " Sep 19 08:42:32.467906 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.467778 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-rbac-proxy-sar-config\") pod \"c6f64b97-8ae5-4690-a718-01a24bc5062f\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " Sep 19 08:42:32.467906 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.467877 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f64b97-8ae5-4690-a718-01a24bc5062f-proxy-tls\") pod \"c6f64b97-8ae5-4690-a718-01a24bc5062f\" (UID: \"c6f64b97-8ae5-4690-a718-01a24bc5062f\") " Sep 19 08:42:32.468074 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.468046 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c6f64b97-8ae5-4690-a718-01a24bc5062f" (UID: "c6f64b97-8ae5-4690-a718-01a24bc5062f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:42:32.468191 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.468109 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:42:32.469786 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.469764 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f64b97-8ae5-4690-a718-01a24bc5062f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c6f64b97-8ae5-4690-a718-01a24bc5062f" (UID: "c6f64b97-8ae5-4690-a718-01a24bc5062f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:42:32.469786 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.469776 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-api-access-5nnnh" (OuterVolumeSpecName: "kube-api-access-5nnnh") pod "c6f64b97-8ae5-4690-a718-01a24bc5062f" (UID: "c6f64b97-8ae5-4690-a718-01a24bc5062f"). InnerVolumeSpecName "kube-api-access-5nnnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 08:42:32.569239 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.569193 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c6f64b97-8ae5-4690-a718-01a24bc5062f-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:42:32.569239 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:32.569232 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nnnh\" (UniqueName: \"kubernetes.io/projected/c6f64b97-8ae5-4690-a718-01a24bc5062f-kube-api-access-5nnnh\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:42:33.311848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:33.311818 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" Sep 19 08:42:33.311848 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:33.311838 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d" event={"ID":"c6f64b97-8ae5-4690-a718-01a24bc5062f","Type":"ContainerDied","Data":"0e5020cd0ec05badedbca2499d68e610d030d14677255788a623295321c9d57a"} Sep 19 08:42:33.312375 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:33.311900 2585 scope.go:117] "RemoveContainer" containerID="e171cc98831c716cbf37fa8c7ae031b3073835eaa77264a4e2a867927f7c1f0c" Sep 19 08:42:33.323219 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:33.323200 2585 scope.go:117] "RemoveContainer" containerID="94375871756e902fa1507c74cae475f0f6c2640c2d1e756afbd52418842bdb55" Sep 19 08:42:33.329342 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:33.329322 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d"] Sep 19 08:42:33.332476 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:33.332454 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5cf0a-predictor-b8c7f7c79-ljd2d"] Sep 19 08:42:35.305438 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:35.305405 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" path="/var/lib/kubelet/pods/c6f64b97-8ae5-4690-a718-01a24bc5062f/volumes" Sep 19 08:42:36.426993 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:36.426951 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:42:37.312177 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:37.312152 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:42:37.312725 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:37.312695 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 19 08:42:38.116570 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:38.116538 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:42:41.427461 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:41.427421 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:42:41.427939 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:41.427515 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:42:46.426925 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:46.426879 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:42:47.312991 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:47.312950 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 19 08:42:51.427456 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:51.427415 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:42:56.426909 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:56.426830 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:42:57.313588 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:57.313549 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 19 08:42:58.170745 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.170705 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr"] Sep 19 08:42:58.171176 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.171144 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kube-rbac-proxy" Sep 19 08:42:58.171176 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.171163 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kube-rbac-proxy" Sep 19 08:42:58.171288 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.171177 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" Sep 19 08:42:58.171288 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.171185 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" Sep 19 08:42:58.171288 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.171265 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kube-rbac-proxy" Sep 19 08:42:58.171288 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.171280 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6f64b97-8ae5-4690-a718-01a24bc5062f" containerName="kserve-container" Sep 19 08:42:58.175662 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.175642 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:58.180336 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.180314 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-88665-serving-cert\"" Sep 19 08:42:58.181080 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.181060 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-88665-kube-rbac-proxy-sar-config\"" Sep 19 08:42:58.202315 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.202294 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr"] Sep 19 08:42:58.269128 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.269104 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174d2700-4f37-4923-8e4e-4b604a1080d2-openshift-service-ca-bundle\") pod \"ensemble-graph-88665-dc77848db-z9wdr\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:58.269261 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.269141 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls\") pod \"ensemble-graph-88665-dc77848db-z9wdr\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:58.369847 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.369814 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174d2700-4f37-4923-8e4e-4b604a1080d2-openshift-service-ca-bundle\") pod \"ensemble-graph-88665-dc77848db-z9wdr\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:58.369999 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.369855 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls\") pod \"ensemble-graph-88665-dc77848db-z9wdr\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:58.369999 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:42:58.369960 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-88665-serving-cert: secret "ensemble-graph-88665-serving-cert" not found Sep 19 08:42:58.370112 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:42:58.370012 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls podName:174d2700-4f37-4923-8e4e-4b604a1080d2 nodeName:}" failed. No retries permitted until 2026-09-19 08:42:58.869998365 +0000 UTC m=+1348.081832200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls") pod "ensemble-graph-88665-dc77848db-z9wdr" (UID: "174d2700-4f37-4923-8e4e-4b604a1080d2") : secret "ensemble-graph-88665-serving-cert" not found Sep 19 08:42:58.370405 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.370385 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174d2700-4f37-4923-8e4e-4b604a1080d2-openshift-service-ca-bundle\") pod \"ensemble-graph-88665-dc77848db-z9wdr\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:58.875237 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.875205 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls\") pod \"ensemble-graph-88665-dc77848db-z9wdr\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:58.879316 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.878402 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls\") pod \"ensemble-graph-88665-dc77848db-z9wdr\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:58.880600 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.880580 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:42:58.976017 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.975986 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fbaa75-267c-4473-bd2e-df43c40ca70c-openshift-service-ca-bundle\") pod \"95fbaa75-267c-4473-bd2e-df43c40ca70c\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " Sep 19 08:42:58.976137 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.976029 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls\") pod \"95fbaa75-267c-4473-bd2e-df43c40ca70c\" (UID: \"95fbaa75-267c-4473-bd2e-df43c40ca70c\") " Sep 19 08:42:58.976332 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.976306 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95fbaa75-267c-4473-bd2e-df43c40ca70c-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "95fbaa75-267c-4473-bd2e-df43c40ca70c" (UID: "95fbaa75-267c-4473-bd2e-df43c40ca70c"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:42:58.977982 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:58.977958 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95fbaa75-267c-4473-bd2e-df43c40ca70c" (UID: "95fbaa75-267c-4473-bd2e-df43c40ca70c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:42:59.077359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.077289 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/95fbaa75-267c-4473-bd2e-df43c40ca70c-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:42:59.077359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.077316 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95fbaa75-267c-4473-bd2e-df43c40ca70c-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:42:59.085247 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.085224 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:42:59.353178 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.353108 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr"] Sep 19 08:42:59.356660 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:42:59.356636 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod174d2700_4f37_4923_8e4e_4b604a1080d2.slice/crio-909bf2941a3ac13f126c711657555900b383403300b8e4c496184d54418bf644 WatchSource:0}: Error finding container 909bf2941a3ac13f126c711657555900b383403300b8e4c496184d54418bf644: Status 404 returned error can't find the container with id 909bf2941a3ac13f126c711657555900b383403300b8e4c496184d54418bf644 Sep 19 08:42:59.407374 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.407349 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" event={"ID":"174d2700-4f37-4923-8e4e-4b604a1080d2","Type":"ContainerStarted","Data":"909bf2941a3ac13f126c711657555900b383403300b8e4c496184d54418bf644"} Sep 19 08:42:59.408585 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.408557 2585 generic.go:356] "Generic (PLEG): container finished" podID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerID="06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96" exitCode=0 Sep 19 08:42:59.408681 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.408596 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" event={"ID":"95fbaa75-267c-4473-bd2e-df43c40ca70c","Type":"ContainerDied","Data":"06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96"} Sep 19 08:42:59.408681 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.408636 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" event={"ID":"95fbaa75-267c-4473-bd2e-df43c40ca70c","Type":"ContainerDied","Data":"d5b2d9d08fc263588d8280916522b17f2b0329d4ff18c64efab4a467760033d2"} Sep 19 08:42:59.408681 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.408663 2585 scope.go:117] "RemoveContainer" containerID="06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96" Sep 19 08:42:59.408681 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.408598 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z" Sep 19 08:42:59.416637 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.416612 2585 scope.go:117] "RemoveContainer" containerID="06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96" Sep 19 08:42:59.416945 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:42:59.416928 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96\": container with ID starting with 06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96 not found: ID does not exist" containerID="06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96" Sep 19 08:42:59.417011 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.416954 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96"} err="failed to get container status \"06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96\": rpc error: code = NotFound desc = could not find container \"06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96\": container with ID starting with 06b34c83515f93a74603006d266f15721a22dbe9a7ac0804673b4e1345f8ce96 not found: ID does not exist" Sep 19 08:42:59.424345 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.424319 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z"] Sep 19 08:42:59.427949 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:42:59.427928 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-5cf0a-5bd5db7dc4-x978z"] Sep 19 08:43:00.415117 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:00.415080 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" event={"ID":"174d2700-4f37-4923-8e4e-4b604a1080d2","Type":"ContainerStarted","Data":"95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004"} Sep 19 08:43:00.415475 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:00.415152 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:43:00.434614 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:00.434567 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" podStartSLOduration=2.434550978 podStartE2EDuration="2.434550978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:42:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:00.43281948 +0000 UTC m=+1349.644653329" watchObservedRunningTime="2026-09-19 08:43:00.434550978 +0000 UTC m=+1349.646384836" Sep 19 08:43:01.305459 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:01.305416 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" path="/var/lib/kubelet/pods/95fbaa75-267c-4473-bd2e-df43c40ca70c/volumes" Sep 19 08:43:06.424488 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:06.424457 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:43:07.312743 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:07.312704 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 19 08:43:08.742648 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.742617 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr"] Sep 19 08:43:08.743079 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.742793 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" containerID="cri-o://95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004" gracePeriod=30 Sep 19 08:43:08.847162 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.847130 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n"] Sep 19 08:43:08.847379 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.847353 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" containerID="cri-o://599d32d3ad057ddd38a981e1d25ebee299f4f4111854f877782a5abe1db4f3aa" gracePeriod=30 Sep 19 08:43:08.847472 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.847403 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kube-rbac-proxy" containerID="cri-o://c4f79e7aae0ba5e6a281c7c4eee4bc966386db0ca70aa8b0c224bb2447284cc0" gracePeriod=30 Sep 19 08:43:08.999172 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.999077 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg"] Sep 19 08:43:08.999478 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.999460 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" Sep 19 08:43:08.999582 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.999482 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" Sep 19 08:43:08.999648 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:08.999585 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="95fbaa75-267c-4473-bd2e-df43c40ca70c" containerName="sequence-graph-5cf0a" Sep 19 08:43:09.002634 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.002614 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.005256 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.005232 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fd725-predictor-serving-cert\"" Sep 19 08:43:09.005370 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.005327 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-fd725-kube-rbac-proxy-sar-config\"" Sep 19 08:43:09.012906 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.012884 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg"] Sep 19 08:43:09.056947 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.056920 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.057111 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.056970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx92r\" (UniqueName: \"kubernetes.io/projected/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-api-access-bx92r\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.057111 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.057012 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.158405 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.158356 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.158640 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.158421 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx92r\" (UniqueName: \"kubernetes.io/projected/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-api-access-bx92r\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.158640 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.158450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.158640 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:43:09.158560 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-fd725-predictor-serving-cert: secret "success-200-isvc-fd725-predictor-serving-cert" not found Sep 19 08:43:09.158640 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:43:09.158633 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls podName:7c00d26d-9f69-4523-b8c9-8f8a259766cd nodeName:}" failed. No retries permitted until 2026-09-19 08:43:09.658613282 +0000 UTC m=+1358.870447117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls") pod "success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" (UID: "7c00d26d-9f69-4523-b8c9-8f8a259766cd") : secret "success-200-isvc-fd725-predictor-serving-cert" not found Sep 19 08:43:09.159147 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.159129 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.181273 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.181243 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx92r\" (UniqueName: \"kubernetes.io/projected/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-api-access-bx92r\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.451801 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.451771 2585 generic.go:356] "Generic (PLEG): container finished" podID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerID="c4f79e7aae0ba5e6a281c7c4eee4bc966386db0ca70aa8b0c224bb2447284cc0" exitCode=2 Sep 19 08:43:09.451956 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.451846 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" event={"ID":"497df55c-e6d1-4283-a4c7-c0ec9db45a72","Type":"ContainerDied","Data":"c4f79e7aae0ba5e6a281c7c4eee4bc966386db0ca70aa8b0c224bb2447284cc0"} Sep 19 08:43:09.663774 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.663738 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.666150 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.666124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls\") pod \"success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:09.912054 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:09.912014 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:10.184300 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:10.184270 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg"] Sep 19 08:43:10.187046 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:43:10.187018 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c00d26d_9f69_4523_b8c9_8f8a259766cd.slice/crio-f1cd53a91ac20214bde705ef1f5b9e4c4893722c4a5f791b377c694a6c6f96f7 WatchSource:0}: Error finding container f1cd53a91ac20214bde705ef1f5b9e4c4893722c4a5f791b377c694a6c6f96f7: Status 404 returned error can't find the container with id f1cd53a91ac20214bde705ef1f5b9e4c4893722c4a5f791b377c694a6c6f96f7 Sep 19 08:43:10.457455 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:10.457357 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" event={"ID":"7c00d26d-9f69-4523-b8c9-8f8a259766cd","Type":"ContainerStarted","Data":"36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60"} Sep 19 08:43:10.457620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:10.457458 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" event={"ID":"7c00d26d-9f69-4523-b8c9-8f8a259766cd","Type":"ContainerStarted","Data":"4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db"} Sep 19 08:43:10.457620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:10.457484 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" event={"ID":"7c00d26d-9f69-4523-b8c9-8f8a259766cd","Type":"ContainerStarted","Data":"f1cd53a91ac20214bde705ef1f5b9e4c4893722c4a5f791b377c694a6c6f96f7"} Sep 19 08:43:10.457620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:10.457540 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:10.486008 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:10.485941 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podStartSLOduration=2.485920364 podStartE2EDuration="2.485920364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:10.484869534 +0000 UTC m=+1359.696703394" watchObservedRunningTime="2026-09-19 08:43:10.485920364 +0000 UTC m=+1359.697754223" Sep 19 08:43:11.423265 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:11.423216 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:43:11.461337 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:11.461305 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:11.462710 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:11.462683 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 19 08:43:12.467596 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.467563 2585 generic.go:356] "Generic (PLEG): container finished" podID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerID="599d32d3ad057ddd38a981e1d25ebee299f4f4111854f877782a5abe1db4f3aa" exitCode=0 Sep 19 08:43:12.467987 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.467639 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" event={"ID":"497df55c-e6d1-4283-a4c7-c0ec9db45a72","Type":"ContainerDied","Data":"599d32d3ad057ddd38a981e1d25ebee299f4f4111854f877782a5abe1db4f3aa"} Sep 19 08:43:12.468050 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.468025 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 19 08:43:12.492979 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.492958 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:43:12.588781 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.588682 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls\") pod \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " Sep 19 08:43:12.588947 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.588797 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-rbac-proxy-sar-config\") pod \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " Sep 19 08:43:12.588947 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.588838 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w48tg\" (UniqueName: \"kubernetes.io/projected/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-api-access-w48tg\") pod \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\" (UID: \"497df55c-e6d1-4283-a4c7-c0ec9db45a72\") " Sep 19 08:43:12.589221 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.589185 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "497df55c-e6d1-4283-a4c7-c0ec9db45a72" (UID: "497df55c-e6d1-4283-a4c7-c0ec9db45a72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:43:12.590948 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.590920 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-api-access-w48tg" (OuterVolumeSpecName: "kube-api-access-w48tg") pod "497df55c-e6d1-4283-a4c7-c0ec9db45a72" (UID: "497df55c-e6d1-4283-a4c7-c0ec9db45a72"). InnerVolumeSpecName "kube-api-access-w48tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 08:43:12.590948 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.590941 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "497df55c-e6d1-4283-a4c7-c0ec9db45a72" (UID: "497df55c-e6d1-4283-a4c7-c0ec9db45a72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:43:12.689860 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.689809 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:43:12.689860 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.689850 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w48tg\" (UniqueName: \"kubernetes.io/projected/497df55c-e6d1-4283-a4c7-c0ec9db45a72-kube-api-access-w48tg\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:43:12.689860 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:12.689860 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/497df55c-e6d1-4283-a4c7-c0ec9db45a72-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:43:13.473304 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:13.473268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" event={"ID":"497df55c-e6d1-4283-a4c7-c0ec9db45a72","Type":"ContainerDied","Data":"634f8798d73a11a1be786133c56765e7b8164f7db38df86a581328f2e49fbb56"} Sep 19 08:43:13.473728 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:13.473336 2585 scope.go:117] "RemoveContainer" containerID="c4f79e7aae0ba5e6a281c7c4eee4bc966386db0ca70aa8b0c224bb2447284cc0" Sep 19 08:43:13.473728 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:13.473274 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n" Sep 19 08:43:13.482269 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:13.482240 2585 scope.go:117] "RemoveContainer" containerID="599d32d3ad057ddd38a981e1d25ebee299f4f4111854f877782a5abe1db4f3aa" Sep 19 08:43:13.491688 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:13.491663 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n"] Sep 19 08:43:13.496029 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:13.496005 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-88665-predictor-585866c4c-t4g2n"] Sep 19 08:43:15.305311 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:15.305276 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" path="/var/lib/kubelet/pods/497df55c-e6d1-4283-a4c7-c0ec9db45a72/volumes" Sep 19 08:43:16.426027 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:16.425985 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:43:17.313487 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:17.313452 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused" Sep 19 08:43:17.473164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:17.473135 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:43:17.473766 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:17.473738 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 19 08:43:21.424013 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:21.423979 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:43:21.424374 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:21.424089 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:43:26.423244 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:26.423199 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:43:27.313500 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:27.313472 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:43:27.474058 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:27.474017 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 19 08:43:31.423224 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:31.423178 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:43:36.422980 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:36.422939 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:43:37.474201 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:37.474156 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 19 08:43:38.881892 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.881868 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:43:38.893619 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.893598 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls\") pod \"174d2700-4f37-4923-8e4e-4b604a1080d2\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " Sep 19 08:43:38.893729 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.893635 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174d2700-4f37-4923-8e4e-4b604a1080d2-openshift-service-ca-bundle\") pod \"174d2700-4f37-4923-8e4e-4b604a1080d2\" (UID: \"174d2700-4f37-4923-8e4e-4b604a1080d2\") " Sep 19 08:43:38.893975 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.893948 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/174d2700-4f37-4923-8e4e-4b604a1080d2-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "174d2700-4f37-4923-8e4e-4b604a1080d2" (UID: "174d2700-4f37-4923-8e4e-4b604a1080d2"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:43:38.895590 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.895566 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "174d2700-4f37-4923-8e4e-4b604a1080d2" (UID: "174d2700-4f37-4923-8e4e-4b604a1080d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:43:38.991054 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991016 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c"] Sep 19 08:43:38.991317 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991305 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" Sep 19 08:43:38.991359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991318 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" Sep 19 08:43:38.991359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991330 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" Sep 19 08:43:38.991359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991336 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" Sep 19 08:43:38.991359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991345 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kube-rbac-proxy" Sep 19 08:43:38.991359 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991351 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kube-rbac-proxy" Sep 19 08:43:38.991509 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991393 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kube-rbac-proxy" Sep 19 08:43:38.991509 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991400 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="497df55c-e6d1-4283-a4c7-c0ec9db45a72" containerName="kserve-container" Sep 19 08:43:38.991509 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.991406 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerName="ensemble-graph-88665" Sep 19 08:43:38.994156 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.994139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:38.994430 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.994403 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/174d2700-4f37-4923-8e4e-4b604a1080d2-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:43:38.994430 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.994431 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/174d2700-4f37-4923-8e4e-4b604a1080d2-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:43:38.997671 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.997623 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3bcba-serving-cert\"" Sep 19 08:43:38.997783 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:38.997767 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3bcba-kube-rbac-proxy-sar-config\"" Sep 19 08:43:39.002658 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.002636 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c"] Sep 19 08:43:39.095007 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.094975 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f00b661-bf48-4768-abfe-fb4710be37de-proxy-tls\") pod \"sequence-graph-3bcba-655868c668-r9k5c\" (UID: \"2f00b661-bf48-4768-abfe-fb4710be37de\") " pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:39.095160 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.095034 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f00b661-bf48-4768-abfe-fb4710be37de-openshift-service-ca-bundle\") pod \"sequence-graph-3bcba-655868c668-r9k5c\" (UID: \"2f00b661-bf48-4768-abfe-fb4710be37de\") " pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:39.195344 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.195312 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f00b661-bf48-4768-abfe-fb4710be37de-openshift-service-ca-bundle\") pod \"sequence-graph-3bcba-655868c668-r9k5c\" (UID: \"2f00b661-bf48-4768-abfe-fb4710be37de\") " pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:39.195558 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.195369 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f00b661-bf48-4768-abfe-fb4710be37de-proxy-tls\") pod \"sequence-graph-3bcba-655868c668-r9k5c\" (UID: \"2f00b661-bf48-4768-abfe-fb4710be37de\") " pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:39.196071 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.196053 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f00b661-bf48-4768-abfe-fb4710be37de-openshift-service-ca-bundle\") pod \"sequence-graph-3bcba-655868c668-r9k5c\" (UID: \"2f00b661-bf48-4768-abfe-fb4710be37de\") " pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:39.197796 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.197778 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f00b661-bf48-4768-abfe-fb4710be37de-proxy-tls\") pod \"sequence-graph-3bcba-655868c668-r9k5c\" (UID: \"2f00b661-bf48-4768-abfe-fb4710be37de\") " pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:39.304918 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.304826 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:39.574333 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.574252 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c"] Sep 19 08:43:39.576825 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.576798 2585 generic.go:356] "Generic (PLEG): container finished" podID="174d2700-4f37-4923-8e4e-4b604a1080d2" containerID="95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004" exitCode=0 Sep 19 08:43:39.576960 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.576851 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" event={"ID":"174d2700-4f37-4923-8e4e-4b604a1080d2","Type":"ContainerDied","Data":"95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004"} Sep 19 08:43:39.576960 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.576856 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" Sep 19 08:43:39.576960 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.576940 2585 scope.go:117] "RemoveContainer" containerID="95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004" Sep 19 08:43:39.577125 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.576924 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr" event={"ID":"174d2700-4f37-4923-8e4e-4b604a1080d2","Type":"ContainerDied","Data":"909bf2941a3ac13f126c711657555900b383403300b8e4c496184d54418bf644"} Sep 19 08:43:39.577125 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:43:39.577024 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f00b661_bf48_4768_abfe_fb4710be37de.slice/crio-3713d565b1320ed214e783315f585548cc69cefff111980b999390791f3771ee WatchSource:0}: Error finding container 3713d565b1320ed214e783315f585548cc69cefff111980b999390791f3771ee: Status 404 returned error can't find the container with id 3713d565b1320ed214e783315f585548cc69cefff111980b999390791f3771ee Sep 19 08:43:39.586046 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.586027 2585 scope.go:117] "RemoveContainer" containerID="95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004" Sep 19 08:43:39.586293 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:43:39.586276 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004\": container with ID starting with 95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004 not found: ID does not exist" containerID="95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004" Sep 19 08:43:39.586351 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.586306 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004"} err="failed to get container status \"95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004\": rpc error: code = NotFound desc = could not find container \"95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004\": container with ID starting with 95a1d969b38f1fc03a155b2ffbb2bdd0390da0f9abbf94225ff296e75116d004 not found: ID does not exist" Sep 19 08:43:39.594351 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.594330 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr"] Sep 19 08:43:39.598428 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:39.598408 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-88665-dc77848db-z9wdr"] Sep 19 08:43:40.583337 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:40.583298 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" event={"ID":"2f00b661-bf48-4768-abfe-fb4710be37de","Type":"ContainerStarted","Data":"cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b"} Sep 19 08:43:40.583767 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:40.583351 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" event={"ID":"2f00b661-bf48-4768-abfe-fb4710be37de","Type":"ContainerStarted","Data":"3713d565b1320ed214e783315f585548cc69cefff111980b999390791f3771ee"} Sep 19 08:43:40.583767 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:40.583418 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:40.599838 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:40.599789 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" podStartSLOduration=2.599770845 podStartE2EDuration="2.599770845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:40.599159567 +0000 UTC m=+1389.810993424" watchObservedRunningTime="2026-09-19 08:43:40.599770845 +0000 UTC m=+1389.811604702" Sep 19 08:43:41.306816 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:41.306782 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="174d2700-4f37-4923-8e4e-4b604a1080d2" path="/var/lib/kubelet/pods/174d2700-4f37-4923-8e4e-4b604a1080d2/volumes" Sep 19 08:43:46.593115 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:46.593081 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:43:47.473792 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:47.473751 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 19 08:43:49.946282 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:49.946244 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c"] Sep 19 08:43:49.946730 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:49.946597 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" containerID="cri-o://cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b" gracePeriod=30 Sep 19 08:43:50.023587 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.023551 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg"] Sep 19 08:43:50.023843 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.023814 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kube-rbac-proxy" containerID="cri-o://6edf64d816b9b9c3ef910401d835b8be4fa402009f5bd915092efc8df33d2ddc" gracePeriod=30 Sep 19 08:43:50.023989 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.023814 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" containerID="cri-o://555a01d8b994af46fae839c380c9ad5085e05beb1473c8fed038af39990ac5ad" gracePeriod=30 Sep 19 08:43:50.134658 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.134626 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg"] Sep 19 08:43:50.138087 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.138066 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.140400 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.140378 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bdc39-kube-rbac-proxy-sar-config\"" Sep 19 08:43:50.140400 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.140394 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bdc39-predictor-serving-cert\"" Sep 19 08:43:50.146973 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.146951 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg"] Sep 19 08:43:50.183706 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.183671 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.184023 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.183716 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jmk4\" (UniqueName: \"kubernetes.io/projected/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-api-access-5jmk4\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.184023 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.183842 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f3cdbe-72f4-464a-b4a1-80620b06effc-proxy-tls\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.284681 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.284585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.284681 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.284634 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jmk4\" (UniqueName: \"kubernetes.io/projected/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-api-access-5jmk4\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.284937 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.284705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f3cdbe-72f4-464a-b4a1-80620b06effc-proxy-tls\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.285420 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.285392 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.287235 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.287210 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f3cdbe-72f4-464a-b4a1-80620b06effc-proxy-tls\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.292961 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.292938 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jmk4\" (UniqueName: \"kubernetes.io/projected/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-api-access-5jmk4\") pod \"success-200-isvc-bdc39-predictor-697bb5b579-crxjg\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.449286 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.449209 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:50.621741 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.621659 2585 generic.go:356] "Generic (PLEG): container finished" podID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerID="6edf64d816b9b9c3ef910401d835b8be4fa402009f5bd915092efc8df33d2ddc" exitCode=2 Sep 19 08:43:50.621741 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.621731 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" event={"ID":"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e","Type":"ContainerDied","Data":"6edf64d816b9b9c3ef910401d835b8be4fa402009f5bd915092efc8df33d2ddc"} Sep 19 08:43:50.742225 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:50.742203 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg"] Sep 19 08:43:50.744212 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:43:50.744184 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f3cdbe_72f4_464a_b4a1_80620b06effc.slice/crio-03d1b9120a539e0aec3ddbb3383096009ec507190e9e6041eef53245d68a7707 WatchSource:0}: Error finding container 03d1b9120a539e0aec3ddbb3383096009ec507190e9e6041eef53245d68a7707: Status 404 returned error can't find the container with id 03d1b9120a539e0aec3ddbb3383096009ec507190e9e6041eef53245d68a7707 Sep 19 08:43:51.592415 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:51.592373 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:43:51.628412 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:51.628378 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" event={"ID":"63f3cdbe-72f4-464a-b4a1-80620b06effc","Type":"ContainerStarted","Data":"a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896"} Sep 19 08:43:51.628593 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:51.628450 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" event={"ID":"63f3cdbe-72f4-464a-b4a1-80620b06effc","Type":"ContainerStarted","Data":"a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4"} Sep 19 08:43:51.628593 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:51.628479 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:51.628593 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:51.628508 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" event={"ID":"63f3cdbe-72f4-464a-b4a1-80620b06effc","Type":"ContainerStarted","Data":"03d1b9120a539e0aec3ddbb3383096009ec507190e9e6041eef53245d68a7707"} Sep 19 08:43:51.628593 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:51.628573 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:51.630168 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:51.630133 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 19 08:43:51.664052 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:51.664000 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podStartSLOduration=1.663981503 podStartE2EDuration="1.663981503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:43:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:43:51.662798325 +0000 UTC m=+1400.874632183" watchObservedRunningTime="2026-09-19 08:43:51.663981503 +0000 UTC m=+1400.875815415" Sep 19 08:43:52.308040 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:52.307999 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused" Sep 19 08:43:52.633462 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:52.633365 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 19 08:43:53.639624 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.639590 2585 generic.go:356] "Generic (PLEG): container finished" podID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerID="555a01d8b994af46fae839c380c9ad5085e05beb1473c8fed038af39990ac5ad" exitCode=0 Sep 19 08:43:53.639975 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.639640 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" event={"ID":"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e","Type":"ContainerDied","Data":"555a01d8b994af46fae839c380c9ad5085e05beb1473c8fed038af39990ac5ad"} Sep 19 08:43:53.678792 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.678767 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:43:53.712769 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.712742 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-rbac-proxy-sar-config\") pod \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " Sep 19 08:43:53.712934 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.712781 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-655jt\" (UniqueName: \"kubernetes.io/projected/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-api-access-655jt\") pod \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " Sep 19 08:43:53.712934 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.712828 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls\") pod \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\" (UID: \"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e\") " Sep 19 08:43:53.713188 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.713157 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" (UID: "03d4b4f1-60e6-4d89-a40b-4b41bc7b731e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:43:53.714933 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.714902 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" (UID: "03d4b4f1-60e6-4d89-a40b-4b41bc7b731e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:43:53.714933 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.714925 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-api-access-655jt" (OuterVolumeSpecName: "kube-api-access-655jt") pod "03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" (UID: "03d4b4f1-60e6-4d89-a40b-4b41bc7b731e"). InnerVolumeSpecName "kube-api-access-655jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 08:43:53.813909 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.813830 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:43:53.813909 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.813855 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655jt\" (UniqueName: \"kubernetes.io/projected/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-kube-api-access-655jt\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:43:53.813909 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:53.813864 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:43:54.646294 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:54.646259 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" event={"ID":"03d4b4f1-60e6-4d89-a40b-4b41bc7b731e","Type":"ContainerDied","Data":"ebef8b0dc35fc71cfdfddf4a0a78e9edd90fa54e43fdf895b1d03b01531d9683"} Sep 19 08:43:54.646739 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:54.646324 2585 scope.go:117] "RemoveContainer" containerID="6edf64d816b9b9c3ef910401d835b8be4fa402009f5bd915092efc8df33d2ddc" Sep 19 08:43:54.646739 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:54.646276 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg" Sep 19 08:43:54.654624 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:54.654557 2585 scope.go:117] "RemoveContainer" containerID="555a01d8b994af46fae839c380c9ad5085e05beb1473c8fed038af39990ac5ad" Sep 19 08:43:54.667432 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:54.667408 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg"] Sep 19 08:43:54.671051 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:54.671033 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3bcba-predictor-5ddb8f447f-jl8jg"] Sep 19 08:43:55.306476 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:55.306437 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" path="/var/lib/kubelet/pods/03d4b4f1-60e6-4d89-a40b-4b41bc7b731e/volumes" Sep 19 08:43:56.591436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:56.591396 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:43:57.473758 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:57.473725 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 19 08:43:57.638014 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:57.637987 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:43:57.638415 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:43:57.638391 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 19 08:44:01.591815 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:01.591765 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:44:01.592191 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:01.591891 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:44:06.591944 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:06.591904 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:44:07.474368 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:07.474340 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:44:07.639258 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:07.639218 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 19 08:44:11.591792 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:11.591750 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:44:16.591330 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:16.591287 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:44:17.639406 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:17.639360 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 19 08:44:18.995922 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:18.995890 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh"] Sep 19 08:44:18.996284 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:18.996179 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" Sep 19 08:44:18.996284 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:18.996189 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" Sep 19 08:44:18.996284 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:18.996202 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kube-rbac-proxy" Sep 19 08:44:18.996284 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:18.996207 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kube-rbac-proxy" Sep 19 08:44:18.996284 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:18.996260 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kube-rbac-proxy" Sep 19 08:44:18.996284 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:18.996268 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="03d4b4f1-60e6-4d89-a40b-4b41bc7b731e" containerName="kserve-container" Sep 19 08:44:18.999197 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:18.999180 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.001752 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.001728 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fd725-serving-cert\"" Sep 19 08:44:19.001854 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.001808 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-fd725-kube-rbac-proxy-sar-config\"" Sep 19 08:44:19.006968 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.006946 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh"] Sep 19 08:44:19.124481 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.124444 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-openshift-service-ca-bundle\") pod \"ensemble-graph-fd725-59fb7df8dd-sg2jh\" (UID: \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\") " pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.124481 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.124486 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-proxy-tls\") pod \"ensemble-graph-fd725-59fb7df8dd-sg2jh\" (UID: \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\") " pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.224916 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.224879 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-openshift-service-ca-bundle\") pod \"ensemble-graph-fd725-59fb7df8dd-sg2jh\" (UID: \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\") " pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.224916 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.224922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-proxy-tls\") pod \"ensemble-graph-fd725-59fb7df8dd-sg2jh\" (UID: \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\") " pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.225659 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.225634 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-openshift-service-ca-bundle\") pod \"ensemble-graph-fd725-59fb7df8dd-sg2jh\" (UID: \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\") " pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.227364 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.227335 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-proxy-tls\") pod \"ensemble-graph-fd725-59fb7df8dd-sg2jh\" (UID: \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\") " pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.310479 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.310408 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.580085 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.580056 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh"] Sep 19 08:44:19.582489 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:44:19.582463 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8c2250_5f88_47c2_9c21_fa09cdd7495e.slice/crio-293781123aa5e99735af5490625d7c553a4268054399e266bdc16399e464e825 WatchSource:0}: Error finding container 293781123aa5e99735af5490625d7c553a4268054399e266bdc16399e464e825: Status 404 returned error can't find the container with id 293781123aa5e99735af5490625d7c553a4268054399e266bdc16399e464e825 Sep 19 08:44:19.746914 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.746875 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" event={"ID":"cd8c2250-5f88-47c2-9c21-fa09cdd7495e","Type":"ContainerStarted","Data":"6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca"} Sep 19 08:44:19.747077 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.746929 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" event={"ID":"cd8c2250-5f88-47c2-9c21-fa09cdd7495e","Type":"ContainerStarted","Data":"293781123aa5e99735af5490625d7c553a4268054399e266bdc16399e464e825"} Sep 19 08:44:19.747077 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.747002 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:19.761977 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:19.761928 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" podStartSLOduration=1.761915905 podStartE2EDuration="1.761915905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:44:19.760269699 +0000 UTC m=+1428.972103556" watchObservedRunningTime="2026-09-19 08:44:19.761915905 +0000 UTC m=+1428.973749762" Sep 19 08:44:20.112431 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.112405 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:44:20.233754 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.233728 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f00b661-bf48-4768-abfe-fb4710be37de-proxy-tls\") pod \"2f00b661-bf48-4768-abfe-fb4710be37de\" (UID: \"2f00b661-bf48-4768-abfe-fb4710be37de\") " Sep 19 08:44:20.233872 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.233778 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f00b661-bf48-4768-abfe-fb4710be37de-openshift-service-ca-bundle\") pod \"2f00b661-bf48-4768-abfe-fb4710be37de\" (UID: \"2f00b661-bf48-4768-abfe-fb4710be37de\") " Sep 19 08:44:20.234224 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.234201 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f00b661-bf48-4768-abfe-fb4710be37de-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "2f00b661-bf48-4768-abfe-fb4710be37de" (UID: "2f00b661-bf48-4768-abfe-fb4710be37de"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:44:20.235847 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.235829 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f00b661-bf48-4768-abfe-fb4710be37de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f00b661-bf48-4768-abfe-fb4710be37de" (UID: "2f00b661-bf48-4768-abfe-fb4710be37de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:44:20.334790 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.334753 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f00b661-bf48-4768-abfe-fb4710be37de-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:44:20.334948 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.334782 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2f00b661-bf48-4768-abfe-fb4710be37de-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:44:20.752453 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.752423 2585 generic.go:356] "Generic (PLEG): container finished" podID="2f00b661-bf48-4768-abfe-fb4710be37de" containerID="cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b" exitCode=137 Sep 19 08:44:20.752640 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.752464 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" Sep 19 08:44:20.752640 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.752515 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" event={"ID":"2f00b661-bf48-4768-abfe-fb4710be37de","Type":"ContainerDied","Data":"cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b"} Sep 19 08:44:20.752640 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.752582 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c" event={"ID":"2f00b661-bf48-4768-abfe-fb4710be37de","Type":"ContainerDied","Data":"3713d565b1320ed214e783315f585548cc69cefff111980b999390791f3771ee"} Sep 19 08:44:20.752640 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.752603 2585 scope.go:117] "RemoveContainer" containerID="cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b" Sep 19 08:44:20.761427 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.761412 2585 scope.go:117] "RemoveContainer" containerID="cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b" Sep 19 08:44:20.761730 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:44:20.761709 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b\": container with ID starting with cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b not found: ID does not exist" containerID="cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b" Sep 19 08:44:20.761781 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.761738 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b"} err="failed to get container status \"cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b\": rpc error: code = NotFound desc = could not find container \"cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b\": container with ID starting with cc258bbb98bf57d79a2d1e80089f7516bd33cc2269a006ffaae4e84102f7559b not found: ID does not exist" Sep 19 08:44:20.773674 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.773652 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c"] Sep 19 08:44:20.781642 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:20.781619 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3bcba-655868c668-r9k5c"] Sep 19 08:44:21.310505 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:21.310475 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" path="/var/lib/kubelet/pods/2f00b661-bf48-4768-abfe-fb4710be37de/volumes" Sep 19 08:44:25.758891 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:25.758815 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:44:27.638775 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:27.638742 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 19 08:44:37.639163 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:37.639120 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 19 08:44:47.639673 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:44:47.639639 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:45:00.190081 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.190045 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj"] Sep 19 08:45:00.190546 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.190358 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" Sep 19 08:45:00.190546 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.190371 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" Sep 19 08:45:00.190546 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.190430 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f00b661-bf48-4768-abfe-fb4710be37de" containerName="sequence-graph-3bcba" Sep 19 08:45:00.194705 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.194687 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.197230 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.197199 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-bdc39-serving-cert\"" Sep 19 08:45:00.197344 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.197206 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-bdc39-kube-rbac-proxy-sar-config\"" Sep 19 08:45:00.202252 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.202232 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj"] Sep 19 08:45:00.236769 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.236732 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80f9b1ba-8f2f-448b-aef7-165345562351-proxy-tls\") pod \"sequence-graph-bdc39-7569bd466c-fjjkj\" (UID: \"80f9b1ba-8f2f-448b-aef7-165345562351\") " pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.236923 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.236780 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80f9b1ba-8f2f-448b-aef7-165345562351-openshift-service-ca-bundle\") pod \"sequence-graph-bdc39-7569bd466c-fjjkj\" (UID: \"80f9b1ba-8f2f-448b-aef7-165345562351\") " pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.337255 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.337212 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80f9b1ba-8f2f-448b-aef7-165345562351-openshift-service-ca-bundle\") pod \"sequence-graph-bdc39-7569bd466c-fjjkj\" (UID: \"80f9b1ba-8f2f-448b-aef7-165345562351\") " pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.337434 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.337299 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80f9b1ba-8f2f-448b-aef7-165345562351-proxy-tls\") pod \"sequence-graph-bdc39-7569bd466c-fjjkj\" (UID: \"80f9b1ba-8f2f-448b-aef7-165345562351\") " pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.337910 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.337880 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80f9b1ba-8f2f-448b-aef7-165345562351-openshift-service-ca-bundle\") pod \"sequence-graph-bdc39-7569bd466c-fjjkj\" (UID: \"80f9b1ba-8f2f-448b-aef7-165345562351\") " pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.339735 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.339717 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80f9b1ba-8f2f-448b-aef7-165345562351-proxy-tls\") pod \"sequence-graph-bdc39-7569bd466c-fjjkj\" (UID: \"80f9b1ba-8f2f-448b-aef7-165345562351\") " pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.505726 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.505615 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.774270 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.774246 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj"] Sep 19 08:45:00.776888 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:45:00.776862 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80f9b1ba_8f2f_448b_aef7_165345562351.slice/crio-8c19ae96a1c5d8b9e5bd7a5c52ddc68ff4cf6d3bc25c4c65c54fa18c4774cfeb WatchSource:0}: Error finding container 8c19ae96a1c5d8b9e5bd7a5c52ddc68ff4cf6d3bc25c4c65c54fa18c4774cfeb: Status 404 returned error can't find the container with id 8c19ae96a1c5d8b9e5bd7a5c52ddc68ff4cf6d3bc25c4c65c54fa18c4774cfeb Sep 19 08:45:00.904198 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.904155 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" event={"ID":"80f9b1ba-8f2f-448b-aef7-165345562351","Type":"ContainerStarted","Data":"65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226"} Sep 19 08:45:00.904404 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.904215 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" event={"ID":"80f9b1ba-8f2f-448b-aef7-165345562351","Type":"ContainerStarted","Data":"8c19ae96a1c5d8b9e5bd7a5c52ddc68ff4cf6d3bc25c4c65c54fa18c4774cfeb"} Sep 19 08:45:00.904404 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.904300 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:00.920827 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:00.920779 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" podStartSLOduration=0.920762552 podStartE2EDuration="920.762552ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:45:00.919730476 +0000 UTC m=+1470.131564333" watchObservedRunningTime="2026-09-19 08:45:00.920762552 +0000 UTC m=+1470.132596447" Sep 19 08:45:06.914625 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:06.914588 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:45:31.253729 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:31.253697 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:45:31.257224 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:45:31.257200 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:50:31.277037 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:50:31.277005 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:50:31.280567 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:50:31.280543 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:52:34.168322 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.168283 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh"] Sep 19 08:52:34.170808 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.168553 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" containerID="cri-o://6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca" gracePeriod=30 Sep 19 08:52:34.306535 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.306491 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg"] Sep 19 08:52:34.306750 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.306722 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" containerID="cri-o://4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db" gracePeriod=30 Sep 19 08:52:34.306821 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.306776 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kube-rbac-proxy" containerID="cri-o://36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60" gracePeriod=30 Sep 19 08:52:34.471539 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.471494 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4"] Sep 19 08:52:34.474733 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.474716 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.477088 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.477061 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f0ae0-predictor-serving-cert\"" Sep 19 08:52:34.477088 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.477074 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f0ae0-kube-rbac-proxy-sar-config\"" Sep 19 08:52:34.483436 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.483409 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4"] Sep 19 08:52:34.516769 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.516737 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.516956 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.516801 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk787\" (UniqueName: \"kubernetes.io/projected/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-api-access-vk787\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.516956 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.516901 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.617354 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.617312 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk787\" (UniqueName: \"kubernetes.io/projected/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-api-access-vk787\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.617564 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.617367 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.617564 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.617471 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.617672 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:52:34.617595 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-serving-cert: secret "success-200-isvc-f0ae0-predictor-serving-cert" not found Sep 19 08:52:34.617769 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:52:34.617707 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls podName:b8a9675c-82d5-4584-92eb-3a986c530a72 nodeName:}" failed. No retries permitted until 2026-09-19 08:52:35.117684482 +0000 UTC m=+1924.329518332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls") pod "success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" (UID: "b8a9675c-82d5-4584-92eb-3a986c530a72") : secret "success-200-isvc-f0ae0-predictor-serving-cert" not found Sep 19 08:52:34.618206 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.618187 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.625865 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.625835 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk787\" (UniqueName: \"kubernetes.io/projected/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-api-access-vk787\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:34.630570 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.630517 2585 generic.go:356] "Generic (PLEG): container finished" podID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerID="36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60" exitCode=2 Sep 19 08:52:34.630570 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:34.630552 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" event={"ID":"7c00d26d-9f69-4523-b8c9-8f8a259766cd","Type":"ContainerDied","Data":"36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60"} Sep 19 08:52:35.121920 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:35.121862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:35.124338 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:35.124314 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls\") pod \"success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:35.386444 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:35.386331 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:35.660342 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:35.660318 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4"] Sep 19 08:52:35.662279 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:52:35.662252 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a9675c_82d5_4584_92eb_3a986c530a72.slice/crio-3e7e0255769c9d996128441ae1706bb7ae1fe2ad04c82f17ca16820bade1c928 WatchSource:0}: Error finding container 3e7e0255769c9d996128441ae1706bb7ae1fe2ad04c82f17ca16820bade1c928: Status 404 returned error can't find the container with id 3e7e0255769c9d996128441ae1706bb7ae1fe2ad04c82f17ca16820bade1c928 Sep 19 08:52:35.757408 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:35.757368 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:52:36.640189 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:36.640152 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" event={"ID":"b8a9675c-82d5-4584-92eb-3a986c530a72","Type":"ContainerStarted","Data":"74e9dba8c6ef62906ad437d645449672b81de8d362547ff5d74165c4fd2c06c2"} Sep 19 08:52:36.640622 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:36.640244 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" event={"ID":"b8a9675c-82d5-4584-92eb-3a986c530a72","Type":"ContainerStarted","Data":"6be935c062f48a77d7860cb2111ac9ae50ebe3d9b4f87471689d898e56ae4820"} Sep 19 08:52:36.640622 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:36.640267 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" event={"ID":"b8a9675c-82d5-4584-92eb-3a986c530a72","Type":"ContainerStarted","Data":"3e7e0255769c9d996128441ae1706bb7ae1fe2ad04c82f17ca16820bade1c928"} Sep 19 08:52:36.640622 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:36.640429 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:36.640622 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:36.640572 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:36.642057 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:36.642030 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 19 08:52:36.658417 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:36.658372 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podStartSLOduration=2.658358052 podStartE2EDuration="2.658358052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:52:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:52:36.656894454 +0000 UTC m=+1925.868728326" watchObservedRunningTime="2026-09-19 08:52:36.658358052 +0000 UTC m=+1925.870191910" Sep 19 08:52:37.468211 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:37.468168 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused" Sep 19 08:52:37.474778 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:37.474743 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused" Sep 19 08:52:37.644209 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:37.644161 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 19 08:52:38.061496 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.061471 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:52:38.149703 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.149666 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx92r\" (UniqueName: \"kubernetes.io/projected/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-api-access-bx92r\") pod \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " Sep 19 08:52:38.149889 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.149732 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls\") pod \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " Sep 19 08:52:38.149889 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.149813 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-rbac-proxy-sar-config\") pod \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\" (UID: \"7c00d26d-9f69-4523-b8c9-8f8a259766cd\") " Sep 19 08:52:38.150164 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.150138 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7c00d26d-9f69-4523-b8c9-8f8a259766cd" (UID: "7c00d26d-9f69-4523-b8c9-8f8a259766cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:52:38.151806 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.151782 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-api-access-bx92r" (OuterVolumeSpecName: "kube-api-access-bx92r") pod "7c00d26d-9f69-4523-b8c9-8f8a259766cd" (UID: "7c00d26d-9f69-4523-b8c9-8f8a259766cd"). InnerVolumeSpecName "kube-api-access-bx92r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 08:52:38.151876 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.151817 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7c00d26d-9f69-4523-b8c9-8f8a259766cd" (UID: "7c00d26d-9f69-4523-b8c9-8f8a259766cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:52:38.250652 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.250614 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:52:38.250652 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.250646 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx92r\" (UniqueName: \"kubernetes.io/projected/7c00d26d-9f69-4523-b8c9-8f8a259766cd-kube-api-access-bx92r\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:52:38.250652 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.250659 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7c00d26d-9f69-4523-b8c9-8f8a259766cd-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:52:38.649131 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.649097 2585 generic.go:356] "Generic (PLEG): container finished" podID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerID="4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db" exitCode=0 Sep 19 08:52:38.649615 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.649146 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" Sep 19 08:52:38.649615 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.649151 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" event={"ID":"7c00d26d-9f69-4523-b8c9-8f8a259766cd","Type":"ContainerDied","Data":"4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db"} Sep 19 08:52:38.649615 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.649212 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg" event={"ID":"7c00d26d-9f69-4523-b8c9-8f8a259766cd","Type":"ContainerDied","Data":"f1cd53a91ac20214bde705ef1f5b9e4c4893722c4a5f791b377c694a6c6f96f7"} Sep 19 08:52:38.649615 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.649251 2585 scope.go:117] "RemoveContainer" containerID="36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60" Sep 19 08:52:38.658089 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.658069 2585 scope.go:117] "RemoveContainer" containerID="4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db" Sep 19 08:52:38.665274 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.665256 2585 scope.go:117] "RemoveContainer" containerID="36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60" Sep 19 08:52:38.665565 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:52:38.665546 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60\": container with ID starting with 36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60 not found: ID does not exist" containerID="36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60" Sep 19 08:52:38.665612 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.665573 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60"} err="failed to get container status \"36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60\": rpc error: code = NotFound desc = could not find container \"36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60\": container with ID starting with 36ddc38d614434e432ac0a206fd1822a15783ee4f6b829891a41c2f1a33bed60 not found: ID does not exist" Sep 19 08:52:38.665612 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.665588 2585 scope.go:117] "RemoveContainer" containerID="4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db" Sep 19 08:52:38.665807 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:52:38.665789 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db\": container with ID starting with 4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db not found: ID does not exist" containerID="4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db" Sep 19 08:52:38.665859 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.665814 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db"} err="failed to get container status \"4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db\": rpc error: code = NotFound desc = could not find container \"4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db\": container with ID starting with 4e2cc48eb3be313f98a8a69f0a071072ae434f0dc9d8060ea44c6c2ca0b623db not found: ID does not exist" Sep 19 08:52:38.670706 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.670686 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg"] Sep 19 08:52:38.674412 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:38.674394 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-fd725-predictor-8568c5bcc7-2cxmg"] Sep 19 08:52:39.306789 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:39.306755 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" path="/var/lib/kubelet/pods/7c00d26d-9f69-4523-b8c9-8f8a259766cd/volumes" Sep 19 08:52:40.757185 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:40.757150 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:52:42.648384 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:42.648357 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:52:42.648853 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:42.648826 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 19 08:52:45.756728 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:45.756690 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:52:45.757122 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:45.756803 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:52:50.756787 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:50.756744 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:52:52.649741 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:52.649702 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 19 08:52:55.756648 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:52:55.756596 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:00.756783 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:00.756743 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:02.649606 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:02.649563 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 19 08:53:04.312868 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.312836 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:53:04.475837 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.475743 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-openshift-service-ca-bundle\") pod \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\" (UID: \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\") " Sep 19 08:53:04.476010 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.475845 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-proxy-tls\") pod \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\" (UID: \"cd8c2250-5f88-47c2-9c21-fa09cdd7495e\") " Sep 19 08:53:04.476207 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.476179 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "cd8c2250-5f88-47c2-9c21-fa09cdd7495e" (UID: "cd8c2250-5f88-47c2-9c21-fa09cdd7495e"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:53:04.477912 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.477883 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cd8c2250-5f88-47c2-9c21-fa09cdd7495e" (UID: "cd8c2250-5f88-47c2-9c21-fa09cdd7495e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:53:04.576872 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.576832 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:04.576872 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.576869 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cd8c2250-5f88-47c2-9c21-fa09cdd7495e-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:04.761632 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.761508 2585 generic.go:356] "Generic (PLEG): container finished" podID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerID="6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca" exitCode=0 Sep 19 08:53:04.761632 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.761570 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" Sep 19 08:53:04.761632 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.761596 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" event={"ID":"cd8c2250-5f88-47c2-9c21-fa09cdd7495e","Type":"ContainerDied","Data":"6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca"} Sep 19 08:53:04.761864 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.761648 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh" event={"ID":"cd8c2250-5f88-47c2-9c21-fa09cdd7495e","Type":"ContainerDied","Data":"293781123aa5e99735af5490625d7c553a4268054399e266bdc16399e464e825"} Sep 19 08:53:04.761864 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.761676 2585 scope.go:117] "RemoveContainer" containerID="6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca" Sep 19 08:53:04.769844 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.769827 2585 scope.go:117] "RemoveContainer" containerID="6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca" Sep 19 08:53:04.770123 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:53:04.770100 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca\": container with ID starting with 6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca not found: ID does not exist" containerID="6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca" Sep 19 08:53:04.770179 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.770132 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca"} err="failed to get container status \"6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca\": rpc error: code = NotFound desc = could not find container \"6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca\": container with ID starting with 6e85f30b84431f56142288c987e68361f9ade7fb66dc65376d662307626f3dca not found: ID does not exist" Sep 19 08:53:04.782449 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.782424 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh"] Sep 19 08:53:04.787817 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:04.787792 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-fd725-59fb7df8dd-sg2jh"] Sep 19 08:53:05.305774 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:05.305739 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" path="/var/lib/kubelet/pods/cd8c2250-5f88-47c2-9c21-fa09cdd7495e/volumes" Sep 19 08:53:12.648890 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:12.648852 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 19 08:53:15.374944 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.374911 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj"] Sep 19 08:53:15.375389 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.375113 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" containerID="cri-o://65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226" gracePeriod=30 Sep 19 08:53:15.462343 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.462309 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg"] Sep 19 08:53:15.462601 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.462571 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" containerID="cri-o://a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4" gracePeriod=30 Sep 19 08:53:15.462742 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.462594 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kube-rbac-proxy" containerID="cri-o://a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896" gracePeriod=30 Sep 19 08:53:15.595820 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.595781 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4"] Sep 19 08:53:15.596182 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596162 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" Sep 19 08:53:15.596266 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596184 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" Sep 19 08:53:15.596266 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596216 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" Sep 19 08:53:15.596266 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596226 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" Sep 19 08:53:15.596266 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596236 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kube-rbac-proxy" Sep 19 08:53:15.596266 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596245 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kube-rbac-proxy" Sep 19 08:53:15.596508 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596336 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kube-rbac-proxy" Sep 19 08:53:15.596508 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596351 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c00d26d-9f69-4523-b8c9-8f8a259766cd" containerName="kserve-container" Sep 19 08:53:15.596508 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.596359 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8c2250-5f88-47c2-9c21-fa09cdd7495e" containerName="ensemble-graph-fd725" Sep 19 08:53:15.599308 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.599288 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.601756 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.601730 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bde64-kube-rbac-proxy-sar-config\"" Sep 19 08:53:15.601857 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.601841 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bde64-predictor-serving-cert\"" Sep 19 08:53:15.607727 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.607706 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4"] Sep 19 08:53:15.658689 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.658655 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvc8\" (UniqueName: \"kubernetes.io/projected/b942c098-1296-429b-8c66-c824c966ba5b-kube-api-access-2tvc8\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.658851 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.658707 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.658851 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.658793 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b942c098-1296-429b-8c66-c824c966ba5b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.759384 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.759347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b942c098-1296-429b-8c66-c824c966ba5b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.759609 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.759409 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvc8\" (UniqueName: \"kubernetes.io/projected/b942c098-1296-429b-8c66-c824c966ba5b-kube-api-access-2tvc8\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.759609 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.759451 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.759609 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:53:15.759576 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-bde64-predictor-serving-cert: secret "success-200-isvc-bde64-predictor-serving-cert" not found Sep 19 08:53:15.759795 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:53:15.759634 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls podName:b942c098-1296-429b-8c66-c824c966ba5b nodeName:}" failed. No retries permitted until 2026-09-19 08:53:16.259616016 +0000 UTC m=+1965.471449859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls") pod "success-200-isvc-bde64-predictor-f9896d57f-c9jg4" (UID: "b942c098-1296-429b-8c66-c824c966ba5b") : secret "success-200-isvc-bde64-predictor-serving-cert" not found Sep 19 08:53:15.760125 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.760098 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b942c098-1296-429b-8c66-c824c966ba5b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.770012 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.769983 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvc8\" (UniqueName: \"kubernetes.io/projected/b942c098-1296-429b-8c66-c824c966ba5b-kube-api-access-2tvc8\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:15.808739 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.808704 2585 generic.go:356] "Generic (PLEG): container finished" podID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerID="a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896" exitCode=2 Sep 19 08:53:15.808905 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:15.808772 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" event={"ID":"63f3cdbe-72f4-464a-b4a1-80620b06effc","Type":"ContainerDied","Data":"a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896"} Sep 19 08:53:16.263608 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:16.263547 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:16.266116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:16.266092 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls\") pod \"success-200-isvc-bde64-predictor-f9896d57f-c9jg4\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:16.511037 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:16.511000 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:16.782422 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:16.782324 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4"] Sep 19 08:53:16.785680 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:53:16.785647 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb942c098_1296_429b_8c66_c824c966ba5b.slice/crio-2d49c7db5abb77b8dc82e96751a3bc595f98a39987287cf9ea6af0cefdbaeaba WatchSource:0}: Error finding container 2d49c7db5abb77b8dc82e96751a3bc595f98a39987287cf9ea6af0cefdbaeaba: Status 404 returned error can't find the container with id 2d49c7db5abb77b8dc82e96751a3bc595f98a39987287cf9ea6af0cefdbaeaba Sep 19 08:53:16.814640 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:16.814603 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" event={"ID":"b942c098-1296-429b-8c66-c824c966ba5b","Type":"ContainerStarted","Data":"2d49c7db5abb77b8dc82e96751a3bc595f98a39987287cf9ea6af0cefdbaeaba"} Sep 19 08:53:16.912906 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:16.912872 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:17.634239 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:17.634190 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused" Sep 19 08:53:17.638637 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:17.638609 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused" Sep 19 08:53:17.820753 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:17.820713 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" event={"ID":"b942c098-1296-429b-8c66-c824c966ba5b","Type":"ContainerStarted","Data":"a053881f2162fb3d73a36595dc1c39e833d380670e2499d1e468d76007bd8860"} Sep 19 08:53:17.820946 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:17.820784 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" event={"ID":"b942c098-1296-429b-8c66-c824c966ba5b","Type":"ContainerStarted","Data":"2b1041df744e13ebb016a966727c6d71749df0ac61cc59628ac4988d5cbf234f"} Sep 19 08:53:17.821016 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:17.820938 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:17.821016 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:17.821000 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:17.822221 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:17.822193 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 19 08:53:17.839632 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:17.839579 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podStartSLOduration=2.839561821 podStartE2EDuration="2.839561821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:53:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:53:17.838379068 +0000 UTC m=+1967.050212919" watchObservedRunningTime="2026-09-19 08:53:17.839561821 +0000 UTC m=+1967.051395677" Sep 19 08:53:18.825589 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:18.825545 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 19 08:53:19.320375 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.320350 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:53:19.389795 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.389713 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jmk4\" (UniqueName: \"kubernetes.io/projected/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-api-access-5jmk4\") pod \"63f3cdbe-72f4-464a-b4a1-80620b06effc\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " Sep 19 08:53:19.389795 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.389751 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f3cdbe-72f4-464a-b4a1-80620b06effc-proxy-tls\") pod \"63f3cdbe-72f4-464a-b4a1-80620b06effc\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " Sep 19 08:53:19.389970 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.389811 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-rbac-proxy-sar-config\") pod \"63f3cdbe-72f4-464a-b4a1-80620b06effc\" (UID: \"63f3cdbe-72f4-464a-b4a1-80620b06effc\") " Sep 19 08:53:19.390182 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.390156 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63f3cdbe-72f4-464a-b4a1-80620b06effc" (UID: "63f3cdbe-72f4-464a-b4a1-80620b06effc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:53:19.391849 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.391821 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f3cdbe-72f4-464a-b4a1-80620b06effc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63f3cdbe-72f4-464a-b4a1-80620b06effc" (UID: "63f3cdbe-72f4-464a-b4a1-80620b06effc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:53:19.391959 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.391933 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-api-access-5jmk4" (OuterVolumeSpecName: "kube-api-access-5jmk4") pod "63f3cdbe-72f4-464a-b4a1-80620b06effc" (UID: "63f3cdbe-72f4-464a-b4a1-80620b06effc"). InnerVolumeSpecName "kube-api-access-5jmk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 08:53:19.490540 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.490486 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:19.490540 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.490541 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jmk4\" (UniqueName: \"kubernetes.io/projected/63f3cdbe-72f4-464a-b4a1-80620b06effc-kube-api-access-5jmk4\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:19.490733 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.490556 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63f3cdbe-72f4-464a-b4a1-80620b06effc-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:19.830568 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.830512 2585 generic.go:356] "Generic (PLEG): container finished" podID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerID="a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4" exitCode=0 Sep 19 08:53:19.830568 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.830554 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" event={"ID":"63f3cdbe-72f4-464a-b4a1-80620b06effc","Type":"ContainerDied","Data":"a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4"} Sep 19 08:53:19.831056 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.830579 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" Sep 19 08:53:19.831056 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.830624 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg" event={"ID":"63f3cdbe-72f4-464a-b4a1-80620b06effc","Type":"ContainerDied","Data":"03d1b9120a539e0aec3ddbb3383096009ec507190e9e6041eef53245d68a7707"} Sep 19 08:53:19.831056 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.830651 2585 scope.go:117] "RemoveContainer" containerID="a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896" Sep 19 08:53:19.839123 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.839106 2585 scope.go:117] "RemoveContainer" containerID="a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4" Sep 19 08:53:19.846559 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.846537 2585 scope.go:117] "RemoveContainer" containerID="a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896" Sep 19 08:53:19.846849 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:53:19.846827 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896\": container with ID starting with a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896 not found: ID does not exist" containerID="a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896" Sep 19 08:53:19.846894 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.846858 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896"} err="failed to get container status \"a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896\": rpc error: code = NotFound desc = could not find container \"a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896\": container with ID starting with a9abc8dd51402e396680e257a68544aae8c25f8ace367ebcb137869afdc06896 not found: ID does not exist" Sep 19 08:53:19.846894 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.846877 2585 scope.go:117] "RemoveContainer" containerID="a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4" Sep 19 08:53:19.847143 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:53:19.847123 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4\": container with ID starting with a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4 not found: ID does not exist" containerID="a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4" Sep 19 08:53:19.847193 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.847148 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4"} err="failed to get container status \"a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4\": rpc error: code = NotFound desc = could not find container \"a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4\": container with ID starting with a8fd67c0d5c72a9df473afb2bca7045a826d9edf32b7e5e11cab760e250d14e4 not found: ID does not exist" Sep 19 08:53:19.853181 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.853158 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg"] Sep 19 08:53:19.861222 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:19.861201 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bdc39-predictor-697bb5b579-crxjg"] Sep 19 08:53:21.306224 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:21.306194 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" path="/var/lib/kubelet/pods/63f3cdbe-72f4-464a-b4a1-80620b06effc/volumes" Sep 19 08:53:21.912704 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:21.912668 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:22.649160 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:22.649119 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused" Sep 19 08:53:23.829999 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:23.829922 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:53:23.830429 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:23.830399 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 19 08:53:26.912930 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:26.912891 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:26.913280 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:26.913013 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:53:31.913180 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:31.913133 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:32.650158 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:32.650128 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:53:33.830728 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:33.830691 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 19 08:53:36.912492 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:36.912447 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:41.912641 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:41.912601 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:43.830880 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:43.830838 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 19 08:53:44.426073 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.426002 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj"] Sep 19 08:53:44.426305 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.426292 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" Sep 19 08:53:44.426305 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.426305 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" Sep 19 08:53:44.426387 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.426317 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kube-rbac-proxy" Sep 19 08:53:44.426387 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.426323 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kube-rbac-proxy" Sep 19 08:53:44.426387 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.426368 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kserve-container" Sep 19 08:53:44.426387 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.426375 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="63f3cdbe-72f4-464a-b4a1-80620b06effc" containerName="kube-rbac-proxy" Sep 19 08:53:44.429155 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.429139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:44.431451 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.431428 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f0ae0-kube-rbac-proxy-sar-config\"" Sep 19 08:53:44.431698 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.431681 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-f0ae0-serving-cert\"" Sep 19 08:53:44.437658 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.437636 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj"] Sep 19 08:53:44.491777 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.491745 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906cfc2f-59b5-4a55-a0cd-bb864320df35-proxy-tls\") pod \"splitter-graph-f0ae0-544b788bf4-stkjj\" (UID: \"906cfc2f-59b5-4a55-a0cd-bb864320df35\") " pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:44.491930 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.491791 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906cfc2f-59b5-4a55-a0cd-bb864320df35-openshift-service-ca-bundle\") pod \"splitter-graph-f0ae0-544b788bf4-stkjj\" (UID: \"906cfc2f-59b5-4a55-a0cd-bb864320df35\") " pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:44.592676 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.592619 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906cfc2f-59b5-4a55-a0cd-bb864320df35-openshift-service-ca-bundle\") pod \"splitter-graph-f0ae0-544b788bf4-stkjj\" (UID: \"906cfc2f-59b5-4a55-a0cd-bb864320df35\") " pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:44.592869 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.592739 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906cfc2f-59b5-4a55-a0cd-bb864320df35-proxy-tls\") pod \"splitter-graph-f0ae0-544b788bf4-stkjj\" (UID: \"906cfc2f-59b5-4a55-a0cd-bb864320df35\") " pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:44.593239 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.593217 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906cfc2f-59b5-4a55-a0cd-bb864320df35-openshift-service-ca-bundle\") pod \"splitter-graph-f0ae0-544b788bf4-stkjj\" (UID: \"906cfc2f-59b5-4a55-a0cd-bb864320df35\") " pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:44.595091 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.595069 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906cfc2f-59b5-4a55-a0cd-bb864320df35-proxy-tls\") pod \"splitter-graph-f0ae0-544b788bf4-stkjj\" (UID: \"906cfc2f-59b5-4a55-a0cd-bb864320df35\") " pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:44.740297 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:44.740211 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:45.009785 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.009709 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj"] Sep 19 08:53:45.012616 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:53:45.012582 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906cfc2f_59b5_4a55_a0cd_bb864320df35.slice/crio-531cb2626cec36da216c76b3665444d1e18bdd50867d356c80df022dbd7316da WatchSource:0}: Error finding container 531cb2626cec36da216c76b3665444d1e18bdd50867d356c80df022dbd7316da: Status 404 returned error can't find the container with id 531cb2626cec36da216c76b3665444d1e18bdd50867d356c80df022dbd7316da Sep 19 08:53:45.501088 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.501061 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:53:45.602187 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.602163 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80f9b1ba-8f2f-448b-aef7-165345562351-proxy-tls\") pod \"80f9b1ba-8f2f-448b-aef7-165345562351\" (UID: \"80f9b1ba-8f2f-448b-aef7-165345562351\") " Sep 19 08:53:45.602273 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.602205 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80f9b1ba-8f2f-448b-aef7-165345562351-openshift-service-ca-bundle\") pod \"80f9b1ba-8f2f-448b-aef7-165345562351\" (UID: \"80f9b1ba-8f2f-448b-aef7-165345562351\") " Sep 19 08:53:45.602607 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.602581 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80f9b1ba-8f2f-448b-aef7-165345562351-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "80f9b1ba-8f2f-448b-aef7-165345562351" (UID: "80f9b1ba-8f2f-448b-aef7-165345562351"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:53:45.604193 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.604166 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80f9b1ba-8f2f-448b-aef7-165345562351-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "80f9b1ba-8f2f-448b-aef7-165345562351" (UID: "80f9b1ba-8f2f-448b-aef7-165345562351"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:53:45.702989 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.702959 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/80f9b1ba-8f2f-448b-aef7-165345562351-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:45.702989 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.702985 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80f9b1ba-8f2f-448b-aef7-165345562351-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:45.930754 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.930706 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" event={"ID":"906cfc2f-59b5-4a55-a0cd-bb864320df35","Type":"ContainerStarted","Data":"a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688"} Sep 19 08:53:45.930933 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.930783 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" event={"ID":"906cfc2f-59b5-4a55-a0cd-bb864320df35","Type":"ContainerStarted","Data":"531cb2626cec36da216c76b3665444d1e18bdd50867d356c80df022dbd7316da"} Sep 19 08:53:45.930933 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.930816 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:45.931813 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.931787 2585 generic.go:356] "Generic (PLEG): container finished" podID="80f9b1ba-8f2f-448b-aef7-165345562351" containerID="65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226" exitCode=137 Sep 19 08:53:45.931905 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.931823 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" Sep 19 08:53:45.931905 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.931825 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" event={"ID":"80f9b1ba-8f2f-448b-aef7-165345562351","Type":"ContainerDied","Data":"65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226"} Sep 19 08:53:45.931905 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.931867 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj" event={"ID":"80f9b1ba-8f2f-448b-aef7-165345562351","Type":"ContainerDied","Data":"8c19ae96a1c5d8b9e5bd7a5c52ddc68ff4cf6d3bc25c4c65c54fa18c4774cfeb"} Sep 19 08:53:45.931905 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.931896 2585 scope.go:117] "RemoveContainer" containerID="65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226" Sep 19 08:53:45.940158 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.940140 2585 scope.go:117] "RemoveContainer" containerID="65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226" Sep 19 08:53:45.940402 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:53:45.940382 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226\": container with ID starting with 65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226 not found: ID does not exist" containerID="65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226" Sep 19 08:53:45.940442 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.940411 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226"} err="failed to get container status \"65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226\": rpc error: code = NotFound desc = could not find container \"65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226\": container with ID starting with 65edb11bab2dcc2308390525512de2fb7b0acffb7df039e2510252e41c05f226 not found: ID does not exist" Sep 19 08:53:45.947051 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.947012 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" podStartSLOduration=1.946998982 podStartE2EDuration="1.946998982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:53:45.945335843 +0000 UTC m=+1995.157169699" watchObservedRunningTime="2026-09-19 08:53:45.946998982 +0000 UTC m=+1995.158832838" Sep 19 08:53:45.956907 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.956860 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj"] Sep 19 08:53:45.958890 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:45.958871 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-bdc39-7569bd466c-fjjkj"] Sep 19 08:53:47.305835 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:47.305802 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" path="/var/lib/kubelet/pods/80f9b1ba-8f2f-448b-aef7-165345562351/volumes" Sep 19 08:53:51.942575 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:51.942539 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:53:53.830561 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:53.830504 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 19 08:53:55.345208 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.345177 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj"] Sep 19 08:53:55.345626 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.345342 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" containerID="cri-o://a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688" gracePeriod=30 Sep 19 08:53:55.432000 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.431967 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4"] Sep 19 08:53:55.432229 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.432192 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" containerID="cri-o://6be935c062f48a77d7860cb2111ac9ae50ebe3d9b4f87471689d898e56ae4820" gracePeriod=30 Sep 19 08:53:55.432360 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.432232 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kube-rbac-proxy" containerID="cri-o://74e9dba8c6ef62906ad437d645449672b81de8d362547ff5d74165c4fd2c06c2" gracePeriod=30 Sep 19 08:53:55.559214 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.559183 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg"] Sep 19 08:53:55.559568 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.559552 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" Sep 19 08:53:55.559662 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.559570 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" Sep 19 08:53:55.559721 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.559665 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f9b1ba-8f2f-448b-aef7-165345562351" containerName="sequence-graph-bdc39" Sep 19 08:53:55.562591 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.562571 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.565211 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.565190 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-de19a-predictor-serving-cert\"" Sep 19 08:53:55.565316 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.565239 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-de19a-kube-rbac-proxy-sar-config\"" Sep 19 08:53:55.571267 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.571247 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg"] Sep 19 08:53:55.687734 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.687705 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz694\" (UniqueName: \"kubernetes.io/projected/91f1ab76-af60-476d-8952-a7f3799f325d-kube-api-access-dz694\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.687734 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.687740 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f1ab76-af60-476d-8952-a7f3799f325d-proxy-tls\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.687990 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.687796 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f1ab76-af60-476d-8952-a7f3799f325d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.788824 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.788782 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz694\" (UniqueName: \"kubernetes.io/projected/91f1ab76-af60-476d-8952-a7f3799f325d-kube-api-access-dz694\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.788996 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.788836 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f1ab76-af60-476d-8952-a7f3799f325d-proxy-tls\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.788996 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.788923 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f1ab76-af60-476d-8952-a7f3799f325d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.789721 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.789694 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f1ab76-af60-476d-8952-a7f3799f325d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.791516 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.791494 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f1ab76-af60-476d-8952-a7f3799f325d-proxy-tls\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.796461 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.796430 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz694\" (UniqueName: \"kubernetes.io/projected/91f1ab76-af60-476d-8952-a7f3799f325d-kube-api-access-dz694\") pod \"success-200-isvc-de19a-predictor-7dc8c765df-mfkkg\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.872970 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.872946 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:55.978241 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.978165 2585 generic.go:356] "Generic (PLEG): container finished" podID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerID="74e9dba8c6ef62906ad437d645449672b81de8d362547ff5d74165c4fd2c06c2" exitCode=2 Sep 19 08:53:55.978369 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:55.978233 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" event={"ID":"b8a9675c-82d5-4584-92eb-3a986c530a72","Type":"ContainerDied","Data":"74e9dba8c6ef62906ad437d645449672b81de8d362547ff5d74165c4fd2c06c2"} Sep 19 08:53:56.139743 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:56.139713 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg"] Sep 19 08:53:56.142442 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:53:56.142414 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f1ab76_af60_476d_8952_a7f3799f325d.slice/crio-7c4c46baf2efabf566344738638492534f7db22b07834e6f8dfa5308c160af09 WatchSource:0}: Error finding container 7c4c46baf2efabf566344738638492534f7db22b07834e6f8dfa5308c160af09: Status 404 returned error can't find the container with id 7c4c46baf2efabf566344738638492534f7db22b07834e6f8dfa5308c160af09 Sep 19 08:53:56.940864 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:56.940822 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:53:56.986475 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:56.986430 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" event={"ID":"91f1ab76-af60-476d-8952-a7f3799f325d","Type":"ContainerStarted","Data":"fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7"} Sep 19 08:53:56.986679 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:56.986511 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" event={"ID":"91f1ab76-af60-476d-8952-a7f3799f325d","Type":"ContainerStarted","Data":"1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096"} Sep 19 08:53:56.986679 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:56.986564 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" event={"ID":"91f1ab76-af60-476d-8952-a7f3799f325d","Type":"ContainerStarted","Data":"7c4c46baf2efabf566344738638492534f7db22b07834e6f8dfa5308c160af09"} Sep 19 08:53:56.986823 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:56.986799 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:56.986882 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:56.986863 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:53:56.988067 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:56.988021 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 19 08:53:57.005143 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:57.005088 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podStartSLOduration=2.005072807 podStartE2EDuration="2.005072807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:53:57.004109557 +0000 UTC m=+2006.215943411" watchObservedRunningTime="2026-09-19 08:53:57.005072807 +0000 UTC m=+2006.216906724" Sep 19 08:53:57.644890 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:57.644849 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused" Sep 19 08:53:57.990643 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:57.990560 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 19 08:53:58.997232 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:58.997204 2585 generic.go:356] "Generic (PLEG): container finished" podID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerID="6be935c062f48a77d7860cb2111ac9ae50ebe3d9b4f87471689d898e56ae4820" exitCode=0 Sep 19 08:53:58.997675 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:58.997258 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" event={"ID":"b8a9675c-82d5-4584-92eb-3a986c530a72","Type":"ContainerDied","Data":"6be935c062f48a77d7860cb2111ac9ae50ebe3d9b4f87471689d898e56ae4820"} Sep 19 08:53:59.075300 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.075275 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:53:59.220789 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.220760 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk787\" (UniqueName: \"kubernetes.io/projected/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-api-access-vk787\") pod \"b8a9675c-82d5-4584-92eb-3a986c530a72\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " Sep 19 08:53:59.220789 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.220795 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-rbac-proxy-sar-config\") pod \"b8a9675c-82d5-4584-92eb-3a986c530a72\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " Sep 19 08:53:59.221007 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.220844 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls\") pod \"b8a9675c-82d5-4584-92eb-3a986c530a72\" (UID: \"b8a9675c-82d5-4584-92eb-3a986c530a72\") " Sep 19 08:53:59.221218 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.221192 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8a9675c-82d5-4584-92eb-3a986c530a72" (UID: "b8a9675c-82d5-4584-92eb-3a986c530a72"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:53:59.222871 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.222843 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-api-access-vk787" (OuterVolumeSpecName: "kube-api-access-vk787") pod "b8a9675c-82d5-4584-92eb-3a986c530a72" (UID: "b8a9675c-82d5-4584-92eb-3a986c530a72"). InnerVolumeSpecName "kube-api-access-vk787". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 08:53:59.222955 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.222869 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8a9675c-82d5-4584-92eb-3a986c530a72" (UID: "b8a9675c-82d5-4584-92eb-3a986c530a72"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:53:59.322077 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.322050 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8a9675c-82d5-4584-92eb-3a986c530a72-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:59.322077 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.322076 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk787\" (UniqueName: \"kubernetes.io/projected/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-api-access-vk787\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:53:59.322234 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:53:59.322085 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8a9675c-82d5-4584-92eb-3a986c530a72-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:54:00.002487 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:00.002444 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" event={"ID":"b8a9675c-82d5-4584-92eb-3a986c530a72","Type":"ContainerDied","Data":"3e7e0255769c9d996128441ae1706bb7ae1fe2ad04c82f17ca16820bade1c928"} Sep 19 08:54:00.002487 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:00.002466 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4" Sep 19 08:54:00.003027 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:00.002514 2585 scope.go:117] "RemoveContainer" containerID="74e9dba8c6ef62906ad437d645449672b81de8d362547ff5d74165c4fd2c06c2" Sep 19 08:54:00.010715 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:00.010697 2585 scope.go:117] "RemoveContainer" containerID="6be935c062f48a77d7860cb2111ac9ae50ebe3d9b4f87471689d898e56ae4820" Sep 19 08:54:00.019853 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:00.019831 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4"] Sep 19 08:54:00.023688 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:00.023667 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f0ae0-predictor-5cf44cb67c-wxwv4"] Sep 19 08:54:01.305658 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:01.305624 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" path="/var/lib/kubelet/pods/b8a9675c-82d5-4584-92eb-3a986c530a72/volumes" Sep 19 08:54:01.940635 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:01.940595 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:54:02.995946 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:02.995912 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:54:02.996506 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:02.996480 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 19 08:54:03.830763 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:03.830721 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused" Sep 19 08:54:06.941561 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:06.941507 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:54:06.941970 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:06.941634 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:54:11.941510 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:11.941466 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:54:12.996728 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:12.996692 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 19 08:54:13.830671 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:13.830643 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 08:54:16.940937 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:16.940892 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:54:21.941074 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:21.941037 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 08:54:22.997073 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:22.997028 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 19 08:54:25.496720 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.496693 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:54:25.526710 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.526684 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906cfc2f-59b5-4a55-a0cd-bb864320df35-proxy-tls\") pod \"906cfc2f-59b5-4a55-a0cd-bb864320df35\" (UID: \"906cfc2f-59b5-4a55-a0cd-bb864320df35\") " Sep 19 08:54:25.526857 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.526775 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906cfc2f-59b5-4a55-a0cd-bb864320df35-openshift-service-ca-bundle\") pod \"906cfc2f-59b5-4a55-a0cd-bb864320df35\" (UID: \"906cfc2f-59b5-4a55-a0cd-bb864320df35\") " Sep 19 08:54:25.527116 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.527092 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906cfc2f-59b5-4a55-a0cd-bb864320df35-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "906cfc2f-59b5-4a55-a0cd-bb864320df35" (UID: "906cfc2f-59b5-4a55-a0cd-bb864320df35"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 08:54:25.528705 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.528683 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906cfc2f-59b5-4a55-a0cd-bb864320df35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "906cfc2f-59b5-4a55-a0cd-bb864320df35" (UID: "906cfc2f-59b5-4a55-a0cd-bb864320df35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 08:54:25.617343 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617251 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk"] Sep 19 08:54:25.617571 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617557 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kube-rbac-proxy" Sep 19 08:54:25.617620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617572 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kube-rbac-proxy" Sep 19 08:54:25.617620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617587 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" Sep 19 08:54:25.617620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617593 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" Sep 19 08:54:25.617620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617609 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" Sep 19 08:54:25.617620 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617615 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" Sep 19 08:54:25.617789 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617661 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kube-rbac-proxy" Sep 19 08:54:25.617789 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617669 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8a9675c-82d5-4584-92eb-3a986c530a72" containerName="kserve-container" Sep 19 08:54:25.617789 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.617678 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerName="splitter-graph-f0ae0" Sep 19 08:54:25.620531 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.620506 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:25.623075 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.623051 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bde64-serving-cert\"" Sep 19 08:54:25.623198 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.623051 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-bde64-kube-rbac-proxy-sar-config\"" Sep 19 08:54:25.627397 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.627372 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls\") pod \"switch-graph-bde64-88ff7fc4f-f6jrk\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:25.627490 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.627414 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97fc033f-fe40-4682-998a-1ce0627bfa07-openshift-service-ca-bundle\") pod \"switch-graph-bde64-88ff7fc4f-f6jrk\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:25.627490 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.627457 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/906cfc2f-59b5-4a55-a0cd-bb864320df35-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:54:25.627490 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.627469 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906cfc2f-59b5-4a55-a0cd-bb864320df35-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 08:54:25.630417 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.630398 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk"] Sep 19 08:54:25.728421 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.728377 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97fc033f-fe40-4682-998a-1ce0627bfa07-openshift-service-ca-bundle\") pod \"switch-graph-bde64-88ff7fc4f-f6jrk\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:25.728614 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.728478 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls\") pod \"switch-graph-bde64-88ff7fc4f-f6jrk\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:25.728692 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:54:25.728616 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-bde64-serving-cert: secret "switch-graph-bde64-serving-cert" not found Sep 19 08:54:25.728692 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:54:25.728689 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls podName:97fc033f-fe40-4682-998a-1ce0627bfa07 nodeName:}" failed. No retries permitted until 2026-09-19 08:54:26.228667249 +0000 UTC m=+2035.440501097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls") pod "switch-graph-bde64-88ff7fc4f-f6jrk" (UID: "97fc033f-fe40-4682-998a-1ce0627bfa07") : secret "switch-graph-bde64-serving-cert" not found Sep 19 08:54:25.728994 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:25.728973 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97fc033f-fe40-4682-998a-1ce0627bfa07-openshift-service-ca-bundle\") pod \"switch-graph-bde64-88ff7fc4f-f6jrk\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:26.102621 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.102585 2585 generic.go:356] "Generic (PLEG): container finished" podID="906cfc2f-59b5-4a55-a0cd-bb864320df35" containerID="a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688" exitCode=0 Sep 19 08:54:26.102797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.102633 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" Sep 19 08:54:26.102797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.102665 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" event={"ID":"906cfc2f-59b5-4a55-a0cd-bb864320df35","Type":"ContainerDied","Data":"a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688"} Sep 19 08:54:26.102797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.102733 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj" event={"ID":"906cfc2f-59b5-4a55-a0cd-bb864320df35","Type":"ContainerDied","Data":"531cb2626cec36da216c76b3665444d1e18bdd50867d356c80df022dbd7316da"} Sep 19 08:54:26.102797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.102770 2585 scope.go:117] "RemoveContainer" containerID="a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688" Sep 19 08:54:26.111446 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.111427 2585 scope.go:117] "RemoveContainer" containerID="a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688" Sep 19 08:54:26.111790 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:54:26.111766 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688\": container with ID starting with a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688 not found: ID does not exist" containerID="a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688" Sep 19 08:54:26.111876 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.111801 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688"} err="failed to get container status \"a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688\": rpc error: code = NotFound desc = could not find container \"a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688\": container with ID starting with a27ceb959e301480e292ed516979ca0f311ba227f339d0d4380747703dda6688 not found: ID does not exist" Sep 19 08:54:26.124552 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.124512 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj"] Sep 19 08:54:26.128023 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.128001 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-f0ae0-544b788bf4-stkjj"] Sep 19 08:54:26.230700 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:26.230658 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls\") pod \"switch-graph-bde64-88ff7fc4f-f6jrk\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:26.230902 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:54:26.230795 2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-bde64-serving-cert: secret "switch-graph-bde64-serving-cert" not found Sep 19 08:54:26.230902 ip-10-0-139-127 kubenswrapper[2585]: E0919 08:54:26.230879 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls podName:97fc033f-fe40-4682-998a-1ce0627bfa07 nodeName:}" failed. No retries permitted until 2026-09-19 08:54:27.230858344 +0000 UTC m=+2036.442692193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls") pod "switch-graph-bde64-88ff7fc4f-f6jrk" (UID: "97fc033f-fe40-4682-998a-1ce0627bfa07") : secret "switch-graph-bde64-serving-cert" not found Sep 19 08:54:27.238729 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:27.238691 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls\") pod \"switch-graph-bde64-88ff7fc4f-f6jrk\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:27.241106 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:27.241085 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls\") pod \"switch-graph-bde64-88ff7fc4f-f6jrk\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:27.307051 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:27.307022 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="906cfc2f-59b5-4a55-a0cd-bb864320df35" path="/var/lib/kubelet/pods/906cfc2f-59b5-4a55-a0cd-bb864320df35/volumes" Sep 19 08:54:27.431252 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:27.431220 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:27.705347 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:27.705325 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk"] Sep 19 08:54:27.707839 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:54:27.707807 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fc033f_fe40_4682_998a_1ce0627bfa07.slice/crio-326d5f9b33d9baef44195e822975287a0fee5bdc756e036ba0b0c3a29c31a6f2 WatchSource:0}: Error finding container 326d5f9b33d9baef44195e822975287a0fee5bdc756e036ba0b0c3a29c31a6f2: Status 404 returned error can't find the container with id 326d5f9b33d9baef44195e822975287a0fee5bdc756e036ba0b0c3a29c31a6f2 Sep 19 08:54:28.116534 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:28.116433 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" event={"ID":"97fc033f-fe40-4682-998a-1ce0627bfa07","Type":"ContainerStarted","Data":"8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7"} Sep 19 08:54:28.116534 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:28.116492 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" event={"ID":"97fc033f-fe40-4682-998a-1ce0627bfa07","Type":"ContainerStarted","Data":"326d5f9b33d9baef44195e822975287a0fee5bdc756e036ba0b0c3a29c31a6f2"} Sep 19 08:54:28.116721 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:28.116557 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:28.133006 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:28.132957 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" podStartSLOduration=3.132941731 podStartE2EDuration="3.132941731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:54:28.132005953 +0000 UTC m=+2037.343839809" watchObservedRunningTime="2026-09-19 08:54:28.132941731 +0000 UTC m=+2037.344775588" Sep 19 08:54:32.997416 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:32.997379 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 19 08:54:34.126256 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:34.126223 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 08:54:42.997049 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:42.997009 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 19 08:54:52.997414 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:54:52.997384 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 08:55:05.585968 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.585891 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875"] Sep 19 08:55:05.589066 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.589050 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:05.591641 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.591621 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-de19a-kube-rbac-proxy-sar-config\"" Sep 19 08:55:05.591751 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.591622 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-de19a-serving-cert\"" Sep 19 08:55:05.598404 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.598377 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875"] Sep 19 08:55:05.747892 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.747857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-openshift-service-ca-bundle\") pod \"splitter-graph-de19a-6cb8f4d8dc-zh875\" (UID: \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\") " pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:05.748059 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.747961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-proxy-tls\") pod \"splitter-graph-de19a-6cb8f4d8dc-zh875\" (UID: \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\") " pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:05.849252 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.849156 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-proxy-tls\") pod \"splitter-graph-de19a-6cb8f4d8dc-zh875\" (UID: \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\") " pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:05.849252 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.849200 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-openshift-service-ca-bundle\") pod \"splitter-graph-de19a-6cb8f4d8dc-zh875\" (UID: \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\") " pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:05.849866 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.849846 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-openshift-service-ca-bundle\") pod \"splitter-graph-de19a-6cb8f4d8dc-zh875\" (UID: \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\") " pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:05.851668 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.851646 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-proxy-tls\") pod \"splitter-graph-de19a-6cb8f4d8dc-zh875\" (UID: \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\") " pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:05.899797 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:05.899771 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:06.166866 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:06.166695 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875"] Sep 19 08:55:06.169649 ip-10-0-139-127 kubenswrapper[2585]: W0919 08:55:06.169620 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f528a9_c94a_42f7_a00d_d9354e4c0ada.slice/crio-58216391ba4055870bb01be8385055ee0615aae41c35e2a19961c7369797b935 WatchSource:0}: Error finding container 58216391ba4055870bb01be8385055ee0615aae41c35e2a19961c7369797b935: Status 404 returned error can't find the container with id 58216391ba4055870bb01be8385055ee0615aae41c35e2a19961c7369797b935 Sep 19 08:55:06.260113 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:06.260075 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" event={"ID":"a0f528a9-c94a-42f7-a00d-d9354e4c0ada","Type":"ContainerStarted","Data":"c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75"} Sep 19 08:55:06.260283 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:06.260138 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" event={"ID":"a0f528a9-c94a-42f7-a00d-d9354e4c0ada","Type":"ContainerStarted","Data":"58216391ba4055870bb01be8385055ee0615aae41c35e2a19961c7369797b935"} Sep 19 08:55:06.260283 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:06.260212 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:06.278042 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:06.277989 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" podStartSLOduration=1.277974343 podStartE2EDuration="1.277974343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-19 08:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 08:55:06.276900724 +0000 UTC m=+2075.488734580" watchObservedRunningTime="2026-09-19 08:55:06.277974343 +0000 UTC m=+2075.489808235" Sep 19 08:55:12.269695 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:12.269658 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 08:55:31.299326 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:31.299292 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 08:55:31.304537 ip-10-0-139-127 kubenswrapper[2585]: I0919 08:55:31.304492 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 09:00:31.322690 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:00:31.322661 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 09:00:31.334047 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:00:31.334027 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 09:03:20.729763 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:20.729724 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875"] Sep 19 09:03:20.730375 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:20.729971 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" containerID="cri-o://c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75" gracePeriod=30 Sep 19 09:03:20.851622 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:20.851587 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg"] Sep 19 09:03:20.851828 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:20.851804 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" containerID="cri-o://1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096" gracePeriod=30 Sep 19 09:03:20.851906 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:20.851862 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kube-rbac-proxy" containerID="cri-o://fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7" gracePeriod=30 Sep 19 09:03:21.142735 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:21.142703 2585 generic.go:356] "Generic (PLEG): container finished" podID="91f1ab76-af60-476d-8952-a7f3799f325d" containerID="fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7" exitCode=2 Sep 19 09:03:21.142926 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:21.142780 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" event={"ID":"91f1ab76-af60-476d-8952-a7f3799f325d","Type":"ContainerDied","Data":"fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7"} Sep 19 09:03:22.268011 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:22.267973 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:03:22.990987 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:22.990947 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused" Sep 19 09:03:22.996590 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:22.996556 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused" Sep 19 09:03:24.292240 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.292217 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 09:03:24.411332 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.411306 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f1ab76-af60-476d-8952-a7f3799f325d-kube-rbac-proxy-sar-config\") pod \"91f1ab76-af60-476d-8952-a7f3799f325d\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " Sep 19 09:03:24.411488 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.411351 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dz694\" (UniqueName: \"kubernetes.io/projected/91f1ab76-af60-476d-8952-a7f3799f325d-kube-api-access-dz694\") pod \"91f1ab76-af60-476d-8952-a7f3799f325d\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " Sep 19 09:03:24.411488 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.411373 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f1ab76-af60-476d-8952-a7f3799f325d-proxy-tls\") pod \"91f1ab76-af60-476d-8952-a7f3799f325d\" (UID: \"91f1ab76-af60-476d-8952-a7f3799f325d\") " Sep 19 09:03:24.411764 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.411737 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f1ab76-af60-476d-8952-a7f3799f325d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "91f1ab76-af60-476d-8952-a7f3799f325d" (UID: "91f1ab76-af60-476d-8952-a7f3799f325d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 09:03:24.413478 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.413451 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f1ab76-af60-476d-8952-a7f3799f325d-kube-api-access-dz694" (OuterVolumeSpecName: "kube-api-access-dz694") pod "91f1ab76-af60-476d-8952-a7f3799f325d" (UID: "91f1ab76-af60-476d-8952-a7f3799f325d"). InnerVolumeSpecName "kube-api-access-dz694". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 09:03:24.413592 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.413498 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91f1ab76-af60-476d-8952-a7f3799f325d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "91f1ab76-af60-476d-8952-a7f3799f325d" (UID: "91f1ab76-af60-476d-8952-a7f3799f325d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 09:03:24.512061 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.512008 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/91f1ab76-af60-476d-8952-a7f3799f325d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:03:24.512061 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.512052 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz694\" (UniqueName: \"kubernetes.io/projected/91f1ab76-af60-476d-8952-a7f3799f325d-kube-api-access-dz694\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:03:24.512061 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:24.512062 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/91f1ab76-af60-476d-8952-a7f3799f325d-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:03:25.160006 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.159976 2585 generic.go:356] "Generic (PLEG): container finished" podID="91f1ab76-af60-476d-8952-a7f3799f325d" containerID="1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096" exitCode=0 Sep 19 09:03:25.160185 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.160025 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" Sep 19 09:03:25.160185 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.160030 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" event={"ID":"91f1ab76-af60-476d-8952-a7f3799f325d","Type":"ContainerDied","Data":"1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096"} Sep 19 09:03:25.160185 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.160139 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg" event={"ID":"91f1ab76-af60-476d-8952-a7f3799f325d","Type":"ContainerDied","Data":"7c4c46baf2efabf566344738638492534f7db22b07834e6f8dfa5308c160af09"} Sep 19 09:03:25.160185 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.160166 2585 scope.go:117] "RemoveContainer" containerID="fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7" Sep 19 09:03:25.168548 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.168432 2585 scope.go:117] "RemoveContainer" containerID="1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096" Sep 19 09:03:25.175878 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.175854 2585 scope.go:117] "RemoveContainer" containerID="fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7" Sep 19 09:03:25.176162 ip-10-0-139-127 kubenswrapper[2585]: E0919 09:03:25.176145 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7\": container with ID starting with fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7 not found: ID does not exist" containerID="fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7" Sep 19 09:03:25.176235 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.176170 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7"} err="failed to get container status \"fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7\": rpc error: code = NotFound desc = could not find container \"fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7\": container with ID starting with fa2e980b1d618f9ea1ddaf896bd38d27155e3b405fcd495f1703072da16b8df7 not found: ID does not exist" Sep 19 09:03:25.176235 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.176188 2585 scope.go:117] "RemoveContainer" containerID="1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096" Sep 19 09:03:25.176449 ip-10-0-139-127 kubenswrapper[2585]: E0919 09:03:25.176431 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096\": container with ID starting with 1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096 not found: ID does not exist" containerID="1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096" Sep 19 09:03:25.176489 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.176456 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096"} err="failed to get container status \"1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096\": rpc error: code = NotFound desc = could not find container \"1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096\": container with ID starting with 1ba3cbeaba26c1c4656a485d38d0303222b8b86c489628ca4e742285f7851096 not found: ID does not exist" Sep 19 09:03:25.182787 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.182761 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg"] Sep 19 09:03:25.189167 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.189142 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-de19a-predictor-7dc8c765df-mfkkg"] Sep 19 09:03:25.305147 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:25.305115 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" path="/var/lib/kubelet/pods/91f1ab76-af60-476d-8952-a7f3799f325d/volumes" Sep 19 09:03:27.268171 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:27.268134 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:03:32.268234 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:32.268193 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:03:32.268645 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:32.268297 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 09:03:37.268150 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:37.268111 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:03:42.267746 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:42.267709 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:03:47.267264 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:47.267224 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:03:50.865672 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:50.865642 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 09:03:51.029696 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.029608 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-openshift-service-ca-bundle\") pod \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\" (UID: \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\") " Sep 19 09:03:51.029879 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.029733 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-proxy-tls\") pod \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\" (UID: \"a0f528a9-c94a-42f7-a00d-d9354e4c0ada\") " Sep 19 09:03:51.029956 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.029929 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "a0f528a9-c94a-42f7-a00d-d9354e4c0ada" (UID: "a0f528a9-c94a-42f7-a00d-d9354e4c0ada"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 09:03:51.031657 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.031636 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a0f528a9-c94a-42f7-a00d-d9354e4c0ada" (UID: "a0f528a9-c94a-42f7-a00d-d9354e4c0ada"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 09:03:51.131221 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.131183 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:03:51.131221 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.131211 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a0f528a9-c94a-42f7-a00d-d9354e4c0ada-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:03:51.256405 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.256372 2585 generic.go:356] "Generic (PLEG): container finished" podID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerID="c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75" exitCode=0 Sep 19 09:03:51.256591 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.256418 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" Sep 19 09:03:51.256591 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.256432 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" event={"ID":"a0f528a9-c94a-42f7-a00d-d9354e4c0ada","Type":"ContainerDied","Data":"c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75"} Sep 19 09:03:51.256591 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.256500 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875" event={"ID":"a0f528a9-c94a-42f7-a00d-d9354e4c0ada","Type":"ContainerDied","Data":"58216391ba4055870bb01be8385055ee0615aae41c35e2a19961c7369797b935"} Sep 19 09:03:51.256591 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.256545 2585 scope.go:117] "RemoveContainer" containerID="c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75" Sep 19 09:03:51.269644 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.269623 2585 scope.go:117] "RemoveContainer" containerID="c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75" Sep 19 09:03:51.269924 ip-10-0-139-127 kubenswrapper[2585]: E0919 09:03:51.269906 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75\": container with ID starting with c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75 not found: ID does not exist" containerID="c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75" Sep 19 09:03:51.269997 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.269932 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75"} err="failed to get container status \"c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75\": rpc error: code = NotFound desc = could not find container \"c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75\": container with ID starting with c82a4a73b3e404af87ffb7120f901bde8cf7a7c4e38e2eb3adb8432d20d1af75 not found: ID does not exist" Sep 19 09:03:51.279949 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.279896 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875"] Sep 19 09:03:51.281639 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.281619 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-de19a-6cb8f4d8dc-zh875"] Sep 19 09:03:51.307287 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:03:51.307259 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" path="/var/lib/kubelet/pods/a0f528a9-c94a-42f7-a00d-d9354e4c0ada/volumes" Sep 19 09:05:31.345252 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:05:31.345153 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 09:05:31.355971 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:05:31.355950 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 09:10:31.366225 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:31.366111 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 09:10:31.378021 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:31.377999 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 09:10:45.592989 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:45.592950 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk"] Sep 19 09:10:45.593355 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:45.593132 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" containerID="cri-o://8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7" gracePeriod=30 Sep 19 09:10:45.724343 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:45.724306 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4"] Sep 19 09:10:45.724571 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:45.724547 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" containerID="cri-o://2b1041df744e13ebb016a966727c6d71749df0ac61cc59628ac4988d5cbf234f" gracePeriod=30 Sep 19 09:10:45.724645 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:45.724582 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kube-rbac-proxy" containerID="cri-o://a053881f2162fb3d73a36595dc1c39e833d380670e2499d1e468d76007bd8860" gracePeriod=30 Sep 19 09:10:46.785318 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:46.785277 2585 generic.go:356] "Generic (PLEG): container finished" podID="b942c098-1296-429b-8c66-c824c966ba5b" containerID="a053881f2162fb3d73a36595dc1c39e833d380670e2499d1e468d76007bd8860" exitCode=2 Sep 19 09:10:46.785758 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:46.785329 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" event={"ID":"b942c098-1296-429b-8c66-c824c966ba5b","Type":"ContainerDied","Data":"a053881f2162fb3d73a36595dc1c39e833d380670e2499d1e468d76007bd8860"} Sep 19 09:10:48.797291 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.797256 2585 generic.go:356] "Generic (PLEG): container finished" podID="b942c098-1296-429b-8c66-c824c966ba5b" containerID="2b1041df744e13ebb016a966727c6d71749df0ac61cc59628ac4988d5cbf234f" exitCode=0 Sep 19 09:10:48.797735 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.797304 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" event={"ID":"b942c098-1296-429b-8c66-c824c966ba5b","Type":"ContainerDied","Data":"2b1041df744e13ebb016a966727c6d71749df0ac61cc59628ac4988d5cbf234f"} Sep 19 09:10:48.857065 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.857041 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 09:10:48.944823 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.944755 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tvc8\" (UniqueName: \"kubernetes.io/projected/b942c098-1296-429b-8c66-c824c966ba5b-kube-api-access-2tvc8\") pod \"b942c098-1296-429b-8c66-c824c966ba5b\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " Sep 19 09:10:48.944823 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.944803 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b942c098-1296-429b-8c66-c824c966ba5b-kube-rbac-proxy-sar-config\") pod \"b942c098-1296-429b-8c66-c824c966ba5b\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " Sep 19 09:10:48.944990 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.944828 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls\") pod \"b942c098-1296-429b-8c66-c824c966ba5b\" (UID: \"b942c098-1296-429b-8c66-c824c966ba5b\") " Sep 19 09:10:48.945128 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.945103 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b942c098-1296-429b-8c66-c824c966ba5b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b942c098-1296-429b-8c66-c824c966ba5b" (UID: "b942c098-1296-429b-8c66-c824c966ba5b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 09:10:48.946815 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.946794 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b942c098-1296-429b-8c66-c824c966ba5b-kube-api-access-2tvc8" (OuterVolumeSpecName: "kube-api-access-2tvc8") pod "b942c098-1296-429b-8c66-c824c966ba5b" (UID: "b942c098-1296-429b-8c66-c824c966ba5b"). InnerVolumeSpecName "kube-api-access-2tvc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 19 09:10:48.946877 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:48.946864 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b942c098-1296-429b-8c66-c824c966ba5b" (UID: "b942c098-1296-429b-8c66-c824c966ba5b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 09:10:49.045548 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.045505 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b942c098-1296-429b-8c66-c824c966ba5b-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:10:49.045548 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.045544 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tvc8\" (UniqueName: \"kubernetes.io/projected/b942c098-1296-429b-8c66-c824c966ba5b-kube-api-access-2tvc8\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:10:49.045548 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.045554 2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b942c098-1296-429b-8c66-c824c966ba5b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:10:49.124711 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.124676 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:10:49.802785 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.802753 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" event={"ID":"b942c098-1296-429b-8c66-c824c966ba5b","Type":"ContainerDied","Data":"2d49c7db5abb77b8dc82e96751a3bc595f98a39987287cf9ea6af0cefdbaeaba"} Sep 19 09:10:49.803203 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.802801 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" Sep 19 09:10:49.803203 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.802845 2585 scope.go:117] "RemoveContainer" containerID="a053881f2162fb3d73a36595dc1c39e833d380670e2499d1e468d76007bd8860" Sep 19 09:10:49.810591 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.810559 2585 scope.go:117] "RemoveContainer" containerID="2b1041df744e13ebb016a966727c6d71749df0ac61cc59628ac4988d5cbf234f" Sep 19 09:10:49.820652 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.820631 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4"] Sep 19 09:10:49.825088 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.825059 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4"] Sep 19 09:10:49.827116 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:49.827090 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bde64-predictor-f9896d57f-c9jg4" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Sep 19 09:10:51.305306 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:51.305273 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b942c098-1296-429b-8c66-c824c966ba5b" path="/var/lib/kubelet/pods/b942c098-1296-429b-8c66-c824c966ba5b/volumes" Sep 19 09:10:54.124038 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:54.124000 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:10:59.124906 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:59.124867 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:10:59.125307 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:10:59.124965 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 09:11:02.153377 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:02.153346 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:03.001173 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:03.001142 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:03.803808 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:03.803778 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:04.124585 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:04.124478 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:11:04.585653 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:04.585622 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:05.372279 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:05.372250 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:06.154474 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:06.154437 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:06.944888 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:06.944857 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:07.758533 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:07.758501 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:08.557494 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:08.557466 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:09.124738 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:09.124698 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:11:09.351299 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:09.351270 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:10.131517 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:10.131482 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:10.931876 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:10.931841 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:11.712980 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:11.712938 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:12.506003 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:12.505971 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:13.322853 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:13.322826 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-bde64-88ff7fc4f-f6jrk_97fc033f-fe40-4682-998a-1ce0627bfa07/switch-graph-bde64/0.log" Sep 19 09:11:14.123989 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:14.123947 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 19 09:11:15.731907 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.731884 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 09:11:15.856036 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.855939 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97fc033f-fe40-4682-998a-1ce0627bfa07-openshift-service-ca-bundle\") pod \"97fc033f-fe40-4682-998a-1ce0627bfa07\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " Sep 19 09:11:15.856036 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.856001 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls\") pod \"97fc033f-fe40-4682-998a-1ce0627bfa07\" (UID: \"97fc033f-fe40-4682-998a-1ce0627bfa07\") " Sep 19 09:11:15.856423 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.856391 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fc033f-fe40-4682-998a-1ce0627bfa07-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "97fc033f-fe40-4682-998a-1ce0627bfa07" (UID: "97fc033f-fe40-4682-998a-1ce0627bfa07"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 19 09:11:15.858064 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.858038 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97fc033f-fe40-4682-998a-1ce0627bfa07" (UID: "97fc033f-fe40-4682-998a-1ce0627bfa07"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 19 09:11:15.895400 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.895359 2585 generic.go:356] "Generic (PLEG): container finished" podID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerID="8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7" exitCode=0 Sep 19 09:11:15.895592 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.895426 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" Sep 19 09:11:15.895592 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.895436 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" event={"ID":"97fc033f-fe40-4682-998a-1ce0627bfa07","Type":"ContainerDied","Data":"8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7"} Sep 19 09:11:15.895592 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.895511 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk" event={"ID":"97fc033f-fe40-4682-998a-1ce0627bfa07","Type":"ContainerDied","Data":"326d5f9b33d9baef44195e822975287a0fee5bdc756e036ba0b0c3a29c31a6f2"} Sep 19 09:11:15.895592 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.895551 2585 scope.go:117] "RemoveContainer" containerID="8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7" Sep 19 09:11:15.903390 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.903370 2585 scope.go:117] "RemoveContainer" containerID="8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7" Sep 19 09:11:15.903719 ip-10-0-139-127 kubenswrapper[2585]: E0919 09:11:15.903697 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7\": container with ID starting with 8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7 not found: ID does not exist" containerID="8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7" Sep 19 09:11:15.903803 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.903728 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7"} err="failed to get container status \"8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7\": rpc error: code = NotFound desc = could not find container \"8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7\": container with ID starting with 8d6e4d9a41c9c3a5a30d1834e1f6c39852b20c9b5249a26ddd52295d02a35dd7 not found: ID does not exist" Sep 19 09:11:15.917085 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.917055 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk"] Sep 19 09:11:15.920438 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.920415 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-bde64-88ff7fc4f-f6jrk"] Sep 19 09:11:15.956797 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.956766 2585 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97fc033f-fe40-4682-998a-1ce0627bfa07-openshift-service-ca-bundle\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:11:15.956797 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:15.956795 2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97fc033f-fe40-4682-998a-1ce0627bfa07-proxy-tls\") on node \"ip-10-0-139-127.ec2.internal\" DevicePath \"\"" Sep 19 09:11:17.305279 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:17.305246 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" path="/var/lib/kubelet/pods/97fc033f-fe40-4682-998a-1ce0627bfa07/volumes" Sep 19 09:11:21.085385 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:21.085356 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mz86w_91530dee-a144-497b-b63a-3b88b4878905/global-pull-secret-syncer/0.log" Sep 19 09:11:21.279582 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:21.279553 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vzdf4_6454de79-4ae5-4e2b-a18d-ac6c4f32c7a7/konnectivity-agent/0.log" Sep 19 09:11:21.302932 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:21.302903 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-127.ec2.internal_a5e2461d28fe7d2289f97533a4da81dc/haproxy/0.log" Sep 19 09:11:25.710697 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:25.710666 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx8mc_3efd392b-dc87-4b0d-baaf-8b88dc2bc206/node-exporter/0.log" Sep 19 09:11:25.731833 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:25.731809 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx8mc_3efd392b-dc87-4b0d-baaf-8b88dc2bc206/kube-rbac-proxy/0.log" Sep 19 09:11:25.758854 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:25.758836 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mx8mc_3efd392b-dc87-4b0d-baaf-8b88dc2bc206/init-textfile/0.log" Sep 19 09:11:28.351764 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:28.351731 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/2.log" Sep 19 09:11:28.360447 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:28.356438 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-fd4bb46c-nvgxc_b88c6ebb-8599-443a-a656-8aa7f8079fd3/console-operator/3.log" Sep 19 09:11:29.309363 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309337 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9"] Sep 19 09:11:29.309656 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309642 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309658 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309668 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309674 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309681 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309688 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309693 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kube-rbac-proxy" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309698 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kube-rbac-proxy" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309704 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" Sep 19 09:11:29.309714 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309709 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" Sep 19 09:11:29.309991 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309719 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kube-rbac-proxy" Sep 19 09:11:29.309991 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309723 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kube-rbac-proxy" Sep 19 09:11:29.309991 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309779 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="97fc033f-fe40-4682-998a-1ce0627bfa07" containerName="switch-graph-bde64" Sep 19 09:11:29.309991 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309786 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kserve-container" Sep 19 09:11:29.309991 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309793 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f1ab76-af60-476d-8952-a7f3799f325d" containerName="kube-rbac-proxy" Sep 19 09:11:29.309991 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309799 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kube-rbac-proxy" Sep 19 09:11:29.309991 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309806 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b942c098-1296-429b-8c66-c824c966ba5b" containerName="kserve-container" Sep 19 09:11:29.309991 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.309810 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0f528a9-c94a-42f7-a00d-d9354e4c0ada" containerName="splitter-graph-de19a" Sep 19 09:11:29.313685 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.313666 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.316251 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.316220 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wb8bw\"/\"kube-root-ca.crt\"" Sep 19 09:11:29.316375 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.316328 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wb8bw\"/\"openshift-service-ca.crt\"" Sep 19 09:11:29.317271 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.317253 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wb8bw\"/\"default-dockercfg-b4dzd\"" Sep 19 09:11:29.322469 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.322448 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9"] Sep 19 09:11:29.406432 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.406374 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5c4db8bdf9-8tzfc_f34f604c-0f98-4be4-91aa-8ec50ac41ecc/volume-data-source-validator/0.log" Sep 19 09:11:29.460040 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.460004 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kjk4\" (UniqueName: \"kubernetes.io/projected/222fb4a7-b12c-451f-8d22-c855e6729258-kube-api-access-9kjk4\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.460040 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.460043 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-sys\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.460283 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.460069 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-lib-modules\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.460283 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.460088 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-podres\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.460283 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.460150 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-proc\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.560950 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.560842 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kjk4\" (UniqueName: \"kubernetes.io/projected/222fb4a7-b12c-451f-8d22-c855e6729258-kube-api-access-9kjk4\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.560950 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.560893 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-sys\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.560950 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.560926 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-lib-modules\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.560950 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.560950 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-podres\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.561238 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.560980 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-proc\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.561238 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.561060 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-sys\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.561238 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.561071 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-proc\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.561238 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.561111 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-lib-modules\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.561238 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.561114 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/222fb4a7-b12c-451f-8d22-c855e6729258-podres\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.568815 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.568792 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kjk4\" (UniqueName: \"kubernetes.io/projected/222fb4a7-b12c-451f-8d22-c855e6729258-kube-api-access-9kjk4\") pod \"perf-node-gather-daemonset-7j8p9\" (UID: \"222fb4a7-b12c-451f-8d22-c855e6729258\") " pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.623644 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.623618 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:29.893020 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.892985 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9"] Sep 19 09:11:29.896566 ip-10-0-139-127 kubenswrapper[2585]: W0919 09:11:29.896535 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod222fb4a7_b12c_451f_8d22_c855e6729258.slice/crio-427e97d209a406b490053a4eac3882e3d56508584245e97ad71a4a51fa144757 WatchSource:0}: Error finding container 427e97d209a406b490053a4eac3882e3d56508584245e97ad71a4a51fa144757: Status 404 returned error can't find the container with id 427e97d209a406b490053a4eac3882e3d56508584245e97ad71a4a51fa144757 Sep 19 09:11:29.949071 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:29.948848 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" event={"ID":"222fb4a7-b12c-451f-8d22-c855e6729258","Type":"ContainerStarted","Data":"427e97d209a406b490053a4eac3882e3d56508584245e97ad71a4a51fa144757"} Sep 19 09:11:30.517385 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:30.517354 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-swsxs_d4fa1e4d-c5fe-4072-8e44-f40975acb77d/dns/0.log" Sep 19 09:11:30.544033 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:30.544000 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-swsxs_d4fa1e4d-c5fe-4072-8e44-f40975acb77d/kube-rbac-proxy/0.log" Sep 19 09:11:30.609939 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:30.609917 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9c7kk_ff2f92e7-7821-4a5c-9807-e81088d7e25f/dns-node-resolver/0.log" Sep 19 09:11:30.953597 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:30.953565 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" event={"ID":"222fb4a7-b12c-451f-8d22-c855e6729258","Type":"ContainerStarted","Data":"2aef96fe875ed960682aaeaf8ca18875a61f64ad9e405207755cb05d076dca26"} Sep 19 09:11:30.953768 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:30.953706 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:30.974025 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:30.973977 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" podStartSLOduration=1.9739633300000001 podStartE2EDuration="1.97396333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-19 09:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-19 09:11:30.972711319 +0000 UTC m=+3060.184545176" watchObservedRunningTime="2026-09-19 09:11:30.97396333 +0000 UTC m=+3060.185797187" Sep 19 09:11:31.108188 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:31.108158 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f5796b8ff-crgk7_050777fb-709c-43e8-a675-0f7f24630a43/registry/0.log" Sep 19 09:11:31.159357 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:31.159325 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mvk4d_492622cf-75d6-416d-bc84-1ba3effd3382/node-ca/0.log" Sep 19 09:11:31.949662 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:31.949639 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-56868b9754-pt5rr_37d3bf81-c5c5-4807-8418-ea8dc81793ca/router/0.log" Sep 19 09:11:32.319582 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:32.319476 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-v68l8_9cd4ad62-dadf-42d7-8696-93041ca4dd12/serve-healthcheck-canary/0.log" Sep 19 09:11:32.662118 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:32.662091 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-268wh_b1e03626-0f6b-41e4-a0c3-9e19c2d2de72/insights-operator/0.log" Sep 19 09:11:32.663023 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:32.663009 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-6f9c644dd9-268wh_b1e03626-0f6b-41e4-a0c3-9e19c2d2de72/insights-operator/1.log" Sep 19 09:11:32.828822 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:32.828789 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fkbpb_f32eb384-ceda-409c-be43-f0f8c014541a/kube-rbac-proxy/0.log" Sep 19 09:11:32.858462 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:32.858438 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fkbpb_f32eb384-ceda-409c-be43-f0f8c014541a/exporter/0.log" Sep 19 09:11:32.883173 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:32.883146 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fkbpb_f32eb384-ceda-409c-be43-f0f8c014541a/extractor/0.log" Sep 19 09:11:35.308673 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:35.308645 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-b2vls_73c79571-bb8e-4e1b-b766-2656b9970517/server/0.log" Sep 19 09:11:35.480016 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:35.479979 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-d45vs_fce49e3c-bddd-4665-a410-5f8fcaca987c/seaweedfs/0.log" Sep 19 09:11:36.966343 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:36.966314 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wb8bw/perf-node-gather-daemonset-7j8p9" Sep 19 09:11:39.693860 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:39.693829 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-srhjm_84b85994-b1ef-49b3-bc7c-dca7867d330b/migrator/0.log" Sep 19 09:11:39.714085 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:39.714057 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-srhjm_84b85994-b1ef-49b3-bc7c-dca7867d330b/graceful-termination/0.log" Sep 19 09:11:41.393160 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.393136 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwwtw_cacb029c-d5ba-4bdf-b8ae-cbd5b809de91/kube-multus-additional-cni-plugins/0.log" Sep 19 09:11:41.418738 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.418714 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwwtw_cacb029c-d5ba-4bdf-b8ae-cbd5b809de91/egress-router-binary-copy/0.log" Sep 19 09:11:41.444946 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.444925 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwwtw_cacb029c-d5ba-4bdf-b8ae-cbd5b809de91/cni-plugins/0.log" Sep 19 09:11:41.465927 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.465903 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwwtw_cacb029c-d5ba-4bdf-b8ae-cbd5b809de91/bond-cni-plugin/0.log" Sep 19 09:11:41.492695 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.492669 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwwtw_cacb029c-d5ba-4bdf-b8ae-cbd5b809de91/routeoverride-cni/0.log" Sep 19 09:11:41.515148 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.515118 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwwtw_cacb029c-d5ba-4bdf-b8ae-cbd5b809de91/whereabouts-cni-bincopy/0.log" Sep 19 09:11:41.539133 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.539105 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-rwwtw_cacb029c-d5ba-4bdf-b8ae-cbd5b809de91/whereabouts-cni/0.log" Sep 19 09:11:41.575116 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.575086 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ndz5k_beb9fb0e-3e64-42d6-a2b1-673a904852bb/kube-multus/0.log" Sep 19 09:11:41.658726 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.658647 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7bd8s_30dd1f4f-1ad8-4cca-94f9-ae57d1e43449/network-metrics-daemon/0.log" Sep 19 09:11:41.680190 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:41.680162 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7bd8s_30dd1f4f-1ad8-4cca-94f9-ae57d1e43449/kube-rbac-proxy/0.log" Sep 19 09:11:42.542391 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:42.542353 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czp6j_c0830ee0-b0ea-4e9d-bac7-3f61cc93712e/ovn-controller/0.log" Sep 19 09:11:42.562966 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:42.562933 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czp6j_c0830ee0-b0ea-4e9d-bac7-3f61cc93712e/ovn-acl-logging/0.log" Sep 19 09:11:42.589374 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:42.589346 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czp6j_c0830ee0-b0ea-4e9d-bac7-3f61cc93712e/kube-rbac-proxy-node/0.log" Sep 19 09:11:42.613211 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:42.613162 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czp6j_c0830ee0-b0ea-4e9d-bac7-3f61cc93712e/kube-rbac-proxy-ovn-metrics/0.log" Sep 19 09:11:42.633337 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:42.633313 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czp6j_c0830ee0-b0ea-4e9d-bac7-3f61cc93712e/northd/0.log" Sep 19 09:11:42.654277 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:42.654251 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czp6j_c0830ee0-b0ea-4e9d-bac7-3f61cc93712e/nbdb/0.log" Sep 19 09:11:42.677332 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:42.677296 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czp6j_c0830ee0-b0ea-4e9d-bac7-3f61cc93712e/sbdb/0.log" Sep 19 09:11:42.721411 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:42.721381 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-czp6j_c0830ee0-b0ea-4e9d-bac7-3f61cc93712e/ovnkube-controller/0.log" Sep 19 09:11:44.153387 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:44.153355 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-l9799_aa486a4a-823d-40ca-b1d1-df39a7a1baf8/network-check-target-container/0.log" Sep 19 09:11:45.060308 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:45.060275 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-jhxfp_7232a07f-d78f-4e05-b33d-9e28c67839e7/iptables-alerter/0.log" Sep 19 09:11:45.720128 ip-10-0-139-127 kubenswrapper[2585]: I0919 09:11:45.720099 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-6nvjv_40a50508-3837-4eea-b588-e34cc16aa8cb/tuned/0.log"