Sep 15 16:26:51.514861 ip-10-0-132-85 systemd[1]: Starting Kubernetes Kubelet...
Sep 15 16:26:51.926806 ip-10-0-132-85 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:51.926806 ip-10-0-132-85 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 15 16:26:51.926806 ip-10-0-132-85 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:51.926806 ip-10-0-132-85 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 15 16:26:51.926806 ip-10-0-132-85 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 15 16:26:51.929166 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.929074    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 15 16:26:51.935502 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935473    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:51.935502 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935497    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:51.935502 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935502    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:51.935502 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935505    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:51.935502 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935509    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935512    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935516    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935519    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935522    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935525    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935527    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935530    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935534    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935536    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935539    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935541    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935546    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935550    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935553    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935555    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935558    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935560    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935563    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:51.935745 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935585    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935589    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935593    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935598    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935601    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935604    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935607    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935610    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935612    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935615    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935617    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935620    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935623    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935626    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935628    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935633    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935635    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935639    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935642    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935644    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:51.936199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935647    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935650    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935653    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935655    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935658    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935660    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935663    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935666    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935669    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935672    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935675    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935685    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935688    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935691    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935693    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935696    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935700    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935704    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935707    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:51.936699 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935710    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935713    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935715    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935718    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935720    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935723    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935726    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935729    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935732    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935735    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935737    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935740    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935742    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935745    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935748    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935751    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935753    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935757    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935760    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935762    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:51.937144 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935765    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935769    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935771    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935774    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935777    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935779    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935782    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935784    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935787    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935789    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935792    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935794    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935797    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935799    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935802    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935805    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.935808    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936524    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936530    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936534    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:51.937646 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936537    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936540    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936543    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936546    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936549    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936551    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936554    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936557    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936562    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936582    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936586    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936589    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936592    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936594    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936598    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936600    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936604    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936607    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936610    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936612    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:51.938153 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936615    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936617    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936620    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936622    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936625    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936627    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936630    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936632    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936635    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936637    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936640    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936645    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936648    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936651    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936653    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936656    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936658    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936661    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936665    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936668    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:51.938669 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936671    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936674    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936676    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936679    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936681    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936684    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936687    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936690    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936693    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936695    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936698    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936701    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936703    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936706    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936709    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936712    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936715    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936717    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936720    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936723    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:51.939150 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936726    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936728    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936731    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936735    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936738    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936741    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936744    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936746    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936749    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936752    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936755    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936757    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936760    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936763    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936765    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936768    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936772    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936775    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936778    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:51.939638 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936781    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936785    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936787    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936790    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936793    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936795    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936798    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936800    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936803    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936805    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936808    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936810    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936813    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936815    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936818    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936820    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.936825    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937256    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937261    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937265    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:51.940103 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937267    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937270    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937273    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937275    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937278    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937281    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937283    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937286    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937291    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937294    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937298    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937301    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937304    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937307    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937310    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937313    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937317    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937320    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937323    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:51.940618 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937326    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937328    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937330    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937333    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937335    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937338    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937341    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937344    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937346    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937349    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937352    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937356    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937360    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937362    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937365    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937367    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937370    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937372    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937375    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:51.941076 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937377    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937380    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937382    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937385    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937387    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937390    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937392    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937395    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937397    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937400    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937402    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937405    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937407    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937410    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937412    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937415    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937418    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937421    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937423    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937426    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:51.941534 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937429    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937432    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937434    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937437    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937440    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937443    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937446    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937448    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937451    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937453    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937456    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937458    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937461    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937463    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937466    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937468    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937471    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937473    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937476    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937478    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:51.942077 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937481    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937485    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937488    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937490    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937493    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937495    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937498    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937500    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937503    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937505    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937508    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937510    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937513    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937515    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937518    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937521    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937523    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.937526    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937625    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937660    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937667    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 15 16:26:51.942558 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937672    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937677    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937680    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937685    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937690    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937693    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937696    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937700    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937703    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937706    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937709    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937712    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937715    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937719    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937722    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937726    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937729    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937732    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937735    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937739    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937743    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937746    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937749    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 15 16:26:51.943084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937752    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937755    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937758    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937761    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937765    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937768    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937772    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937775    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937778    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937780    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937784    2566 flags.go:64] FLAG: --enable-server="true"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937786    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937791    2566 flags.go:64] FLAG: --event-burst="100"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937794    2566 flags.go:64] FLAG: --event-qps="50"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937797    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937800    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937803    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937807    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937810    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937813    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937816    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937819    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937822    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937825    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937828    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 15 16:26:51.943694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937831    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937834    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937836    2566 flags.go:64] FLAG: --feature-gates=""
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937840    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937843    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937846    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937849    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937852    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937858    2566 flags.go:64] FLAG: --help="false"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937861    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-132-85.ec2.internal"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937866    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937869    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937873    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937876    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937880    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937883    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937886    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937889    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937892    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937894    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937897    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937900    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937903    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937906    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 15 16:26:51.944389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937909    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937912    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937915    2566 flags.go:64] FLAG: --lock-file=""
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937918    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937921    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937924    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937929    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937933    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937936    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937939    2566 flags.go:64] FLAG: --logging-format="text"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937941    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937945    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937947    2566 flags.go:64] FLAG: --manifest-url=""
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937950    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937955    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937958    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937963    2566 flags.go:64] FLAG: --max-pods="110"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937967    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937972    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937975    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937978    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937981    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937984    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937988    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937995    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 15 16:26:51.944995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.937998    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938001    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938004    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938008    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938014    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938017    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938020    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938023    2566 flags.go:64] FLAG: --port="10250"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938027    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938030    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0591233dd6c8937a5"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938033    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938036    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938039    2566 flags.go:64] FLAG: --register-node="true"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938042    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938045    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938050    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938054    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938057    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938060    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938063    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938066    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938069    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938072    2566 flags.go:64] FLAG: --runonce="false"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938075    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938079    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 15 16:26:51.945616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938083    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938086    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938089    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938092    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938096    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938099    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938102    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938105    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938108    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938110    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938113    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938116    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938119    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938125    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938128    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938131    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938134    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938137    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938140    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938143    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938146    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938149    2566 flags.go:64] FLAG: --v="2"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938153    2566 flags.go:64] FLAG: --version="false"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938158    2566 flags.go:64] FLAG: --vmodule=""
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938162    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 15 16:26:51.946217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.938165    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938288    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938292    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938294    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938297    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938301    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938305    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938309    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938312    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938315    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938318    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938321    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938324    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938327    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938330    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938333    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938336    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938339    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938342    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938345    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:51.947029 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938347    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938350    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938353    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938355    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938358    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938360    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938363    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938366    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938368    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938370    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938373    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938376    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938379    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938382    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938385    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938389    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938392    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938395    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938400    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:51.947541 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938405    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938409    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938412    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938415    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938418    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938420    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938423    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938425    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938428    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938431    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938434    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938438    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938441    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938445    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938450    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938453    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938455    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938458    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938461    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938464    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:51.948032 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938466    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938469    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938472    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938475    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938477    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938481    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938483    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938486    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938488    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938491    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938494    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938498    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938501    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938504    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938506    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938511    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938514    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938517    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938520    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:51.948520 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938523    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938526    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938529    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938531    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938534    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938537    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938540    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938542    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938545    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938548    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938550    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938553    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938556    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938558    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938560    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938563    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938579    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938583    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938585    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938588    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:51.949005 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938591    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.938594    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.939290    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.946204    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.946224    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.946245    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.946251    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946308    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946315    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946319    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946322    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946325    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946328    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946331    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:51.949507 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946334    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946337    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946339    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946342    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946345    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946347    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946350    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946353    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946357    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946361    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946364    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946366    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946370    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946372    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946375    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946377    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946380    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946384    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946387    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:51.949880 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946390    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946392    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946395    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946399    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946402    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946404    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946407    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946410    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946413    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946415    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946418    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946421    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946424    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946426    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946429    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946432    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946434    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946437    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946439    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946442    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:51.950346 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946445    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946447    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946450    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946452    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946455    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946457    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946461    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946463    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946466    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946468    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946471    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946477    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946480    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946483    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946486    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946489    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946492    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946495    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946498    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:51.950930 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946500    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946503    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946505    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946508    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946510    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946513    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946516    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946519    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946521    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946524    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946526    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946529    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946532    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946534    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946537    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946539    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946542    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946545    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946547    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946550    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:51.951394 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946553    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946555    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946558    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946561    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946563    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946582    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946585    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946587    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946593    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946596    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946600    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946603    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946606    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946609    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:51.951942 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.946614    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946736    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946741    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946744    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946747    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946750    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946753    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946755    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946758    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946760    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946763    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946766    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946769    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946771    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946774    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946777    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946779    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946782    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946784    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 15 16:26:51.952287 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946787    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946790    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946792    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946796    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946800    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946804    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946807    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946810    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946813    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946816    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946819    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946823    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946826    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946829    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946832    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946835    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946837    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946840    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946843    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946846    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 15 16:26:51.952754 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946849    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946851    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946854    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946857    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946859    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946862    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946866    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946869    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946871    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946874    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946876    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946879    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946881    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946884    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946886    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946889    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946891    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946893    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946897    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 15 16:26:51.953238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946900    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946903    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946906    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946908    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946911    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946913    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946916    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946918    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946921    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946923    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946926    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946929    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946931    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946934    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946936    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946939    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946942    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946944    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946947    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946949    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 15 16:26:51.953704 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946952    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946954    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946957    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946960    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946962    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946965    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946967    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946970    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946972    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946975    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946977    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946980    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946983    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946986    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946989    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946992    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946994    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946997    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.946999    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.947002    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 15 16:26:51.954199 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.947004    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:51.947007    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.947012    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.947138    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.949155    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.950177    2566 server.go:1019] "Starting client certificate rotation"
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.950280    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.950321    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.952898    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 16:26:51.954756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.954058    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 15 16:26:51.966994 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.966968    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 15 16:26:51.970171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.970150    2566 log.go:25] "Validated CRI v1 image API"
Sep 15 16:26:51.971100 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.971084    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 15 16:26:51.975590 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.975554    2566 fs.go:135] Filesystem UUIDs: map[15e49264-eeb6-483b-8d2f-b7071a517a83:/dev/nvme0n1p3 1c070938-7cfa-4ccf-ac8d-8d7625f4b258:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 15 16:26:51.975657 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.975589    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 15 16:26:51.981598 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.981463    2566 manager.go:216] Machine: {Timestamp:2026-09-15 16:26:51.980022556 +0000 UTC m=+0.347847749 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098580 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec201de067e340373e0273e939236c4f SystemUUID:ec201de0-67e3-4037-3e02-73e939236c4f BootID:2c9f6a74-50ac-4a29-a191-106e115e6f3a Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f7:8e:9a:b5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f7:8e:9a:b5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1e:d2:a4:60:45:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 15 16:26:51.981598 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.981590    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 15 16:26:51.981766 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.981715    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 15 16:26:51.984493 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.984457    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 15 16:26:51.984649 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.984495    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-85.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 15 16:26:51.985338 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.985328    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 15 16:26:51.985382 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.985341    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 15 16:26:51.985382 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.985356    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 16:26:51.985382 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.985370    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 15 16:26:51.986196 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.986187    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 15 16:26:51.987398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.987381    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 16:26:51.989922 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.989909    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 16:26:51.990059 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.990050    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 15 16:26:51.990097 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.990079    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 15 16:26:51.990097 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.990090    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 15 16:26:51.990149 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.990108    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 15 16:26:51.990149 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.990123    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 15 16:26:51.990149 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.990147    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 15 16:26:51.990502 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.990490    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 16:26:51.990561 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.990507    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 15 16:26:51.993668 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.993650    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 15 16:26:51.995745 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.995731    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 15 16:26:51.995815 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.995761    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 15 16:26:51.998695 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998681    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998701    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998708    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998713    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998721    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998727    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998740    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998745    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998751    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998757    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 15 16:26:51.998759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998765    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 15 16:26:51.999141 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.998770    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 15 16:26:51.999905 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.999894    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 15 16:26:51.999905 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:51.999904    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 15 16:26:52.001021 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.000993    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 15 16:26:52.001021 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.001011    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-85.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 15 16:26:52.002742 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.002727    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-85.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 15 16:26:52.003750 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.003736    2566 server.go:1268] "Started kubelet"
Sep 15 16:26:52.003809 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.003779    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 15 16:26:52.003855 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.003814    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 15 16:26:52.003904 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.003891    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 15 16:26:52.004524 ip-10-0-132-85 systemd[1]: Started Kubernetes Kubelet.
Sep 15 16:26:52.011190 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.010729    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 15 16:26:52.012189 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.012167    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 15 16:26:52.013098 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.013078    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 15 16:26:52.015243 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.015225    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 15 16:26:52.015243 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.015237    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 15 16:26:52.015375 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.015295    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.015557 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.014491    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-85.ec2.internal.18d58b522cbb88cd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-85.ec2.internal,UID:ip-10-0-132-85.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-85.ec2.internal,},FirstTimestamp:2026-09-15 16:26:52.003707085 +0000 UTC m=+0.371532274,LastTimestamp:2026-09-15 16:26:52.003707085 +0000 UTC m=+0.371532274,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-85.ec2.internal,}"
Sep 15 16:26:52.015897 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.015875    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 15 16:26:52.015897 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.015883    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 15 16:26:52.016047 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.015936    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 15 16:26:52.016162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016147    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 15 16:26:52.016162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016160    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 15 16:26:52.016398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016384    2566 factory.go:153] Registering CRI-O factory
Sep 15 16:26:52.016454 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016404    2566 factory.go:223] Registration of the crio container factory successfully
Sep 15 16:26:52.016501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016454    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 15 16:26:52.016501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016464    2566 factory.go:55] Registering systemd factory
Sep 15 16:26:52.016501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016472    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 15 16:26:52.016501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016488    2566 factory.go:103] Registering Raw factory
Sep 15 16:26:52.016710 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016502    2566 manager.go:1196] Started watching for new ooms in manager
Sep 15 16:26:52.016871 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.016858    2566 manager.go:349] Starting recovery of all containers
Sep 15 16:26:52.024190 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.024130    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 15 16:26:52.025364 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.025334    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 15 16:26:52.025806 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.025786    2566 manager.go:354] Recovery completed
Sep 15 16:26:52.030212 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.030193    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-85.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 15 16:26:52.031712 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.031697    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:52.033426 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.033405    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:52.033505 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.033443    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:52.033505 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.033458    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:52.034125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.034108    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 15 16:26:52.034125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.034124    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 15 16:26:52.034245 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.034140    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 15 16:26:52.034245 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.034141    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:52.034977 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.034962    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:52.035045 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.034995    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:52.035045 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.035010    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:52.035555 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.035488    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-85.ec2.internal.18d58b522e8102e7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-85.ec2.internal,UID:ip-10-0-132-85.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-85.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-85.ec2.internal,},FirstTimestamp:2026-09-15 16:26:52.033426151 +0000 UTC m=+0.401251357,LastTimestamp:2026-09-15 16:26:52.033426151 +0000 UTC m=+0.401251357,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-85.ec2.internal,}"
Sep 15 16:26:52.049242 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.049168    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-85.ec2.internal.18d58b522e8160c9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-85.ec2.internal,UID:ip-10-0-132-85.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-85.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-85.ec2.internal,},FirstTimestamp:2026-09-15 16:26:52.033450185 +0000 UTC m=+0.401275375,LastTimestamp:2026-09-15 16:26:52.033450185 +0000 UTC m=+0.401275375,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-85.ec2.internal,}"
Sep 15 16:26:52.059703 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.059632    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-85.ec2.internal.18d58b522e819743  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-85.ec2.internal,UID:ip-10-0-132-85.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-85.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-85.ec2.internal,},FirstTimestamp:2026-09-15 16:26:52.033464131 +0000 UTC m=+0.401289323,LastTimestamp:2026-09-15 16:26:52.033464131 +0000 UTC m=+0.401289323,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-85.ec2.internal,}"
Sep 15 16:26:52.071411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.071388    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxjkh"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.077470    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-132-85.ec2.internal.18d58b522e8102e7\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-85.ec2.internal.18d58b522e8102e7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-85.ec2.internal,UID:ip-10-0-132-85.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-85.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-85.ec2.internal,},FirstTimestamp:2026-09-15 16:26:52.033426151 +0000 UTC m=+0.401251357,LastTimestamp:2026-09-15 16:26:52.034978032 +0000 UTC m=+0.402803227,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-85.ec2.internal,}"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.082556    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pxjkh"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.115661    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.152554    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.152615    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.152636    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.152670    2566 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.155069    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.180230    2566 policy_none.go:49] "None policy: Start"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.180245    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 15 16:26:52.180732 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.180256    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 15 16:26:52.181278 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.181265    2566 policy_none.go:47] "Start"
Sep 15 16:26:52.211180 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.211160    2566 manager.go:337] "Starting Device Plugin manager"
Sep 15 16:26:52.211280 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.211198    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 15 16:26:52.211280 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.211213    2566 server.go:87] "Starting device plugin registration server"
Sep 15 16:26:52.211492 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.211481    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 15 16:26:52.211536 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.211494    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 15 16:26:52.211634 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.211607    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 15 16:26:52.211727 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.211701    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 15 16:26:52.211727 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.211712    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 15 16:26:52.212247 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.212225    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 15 16:26:52.212355 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.212281    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.236727 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.236710    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-132-85.ec2.internal\" not found" node="ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.252782 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.252762    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"]
Sep 15 16:26:52.252842 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.252828    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:52.254331 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.254316    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:52.254389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.254347    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:52.254389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.254358    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:52.255508 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.255496    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:52.255701 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.255645    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.255701 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.255701    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:52.256218 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.256203    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:52.256262 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.256231    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:52.256262 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.256242    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:52.256327 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.256204    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:52.256327 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.256309    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:52.256327 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.256321    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:52.257458 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.257442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.257591 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.257466    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:52.258135 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.258110    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:52.258209 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.258146    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:52.258209 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.258157    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:52.280199 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.280180    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-85.ec2.internal\" not found" node="ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.284511 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.284495    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-85.ec2.internal\" not found" node="ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.312112 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.312090    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:52.313385 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.313369    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:52.313502 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.313399    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:52.313502 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.313408    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:52.313502 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.313430    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.318582 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.318555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37412cca01add92883383fcf18487e21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"37412cca01add92883383fcf18487e21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.318654 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.318596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37412cca01add92883383fcf18487e21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"37412cca01add92883383fcf18487e21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.318654 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.318622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c252aca089984aa62b2a4fbb8eb23028-config\") pod \"kube-apiserver-proxy-ip-10-0-132-85.ec2.internal\" (UID: \"c252aca089984aa62b2a4fbb8eb23028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.328311 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.328296    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.328369 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.328313    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-85.ec2.internal\": node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.390735 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.390713    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.419254 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.419228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37412cca01add92883383fcf18487e21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"37412cca01add92883383fcf18487e21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.419326 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.419260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c252aca089984aa62b2a4fbb8eb23028-config\") pod \"kube-apiserver-proxy-ip-10-0-132-85.ec2.internal\" (UID: \"c252aca089984aa62b2a4fbb8eb23028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.419326 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.419281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37412cca01add92883383fcf18487e21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"37412cca01add92883383fcf18487e21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.419406 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.419330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37412cca01add92883383fcf18487e21-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"37412cca01add92883383fcf18487e21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.419406 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.419391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37412cca01add92883383fcf18487e21-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal\" (UID: \"37412cca01add92883383fcf18487e21\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.419476 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.419421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c252aca089984aa62b2a4fbb8eb23028-config\") pod \"kube-apiserver-proxy-ip-10-0-132-85.ec2.internal\" (UID: \"c252aca089984aa62b2a4fbb8eb23028\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.493198 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.493101    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.586668 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.586628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.587871 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.587848    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:52.593928 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.593911    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.695176 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.695002    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.800860 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.800795    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.903241 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:52.903197    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:52.920483 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.920452    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:52.951582 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.951540    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 15 16:26:52.952011 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.951708    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 16:26:52.952011 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.951709    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 15 16:26:52.990211 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:52.990178    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:53.003340 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:53.003307    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:53.058102 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.058031    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 15 16:26:53.059040 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.059021    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientMemory"
Sep 15 16:26:53.059110 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.059057    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasNoDiskPressure"
Sep 15 16:26:53.059110 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.059068    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeHasSufficientPID"
Sep 15 16:26:53.087341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.087310    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-14 16:21:52 +0000 UTC" deadline="2028-06-17 10:24:10.059050268 +0000 UTC"
Sep 15 16:26:53.087341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.087335    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15377h57m16.971718841s"
Sep 15 16:26:53.103497 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:53.103473    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-85.ec2.internal\" not found"
Sep 15 16:26:53.188710 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.188686    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:53.216210 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.216187    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:53.234660 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.234629    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 16:26:53.235719 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.235702    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal"
Sep 15 16:26:53.247990 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.247972    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 15 16:26:53.300676 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:53.300621    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc252aca089984aa62b2a4fbb8eb23028.slice/crio-6bd7e11b49fc68a3555cc87fe1dd9afe42a1d9d9fff13bc3e1ac61e89c3c4a30 WatchSource:0}: Error finding container 6bd7e11b49fc68a3555cc87fe1dd9afe42a1d9d9fff13bc3e1ac61e89c3c4a30: Status 404 returned error can't find the container with id 6bd7e11b49fc68a3555cc87fe1dd9afe42a1d9d9fff13bc3e1ac61e89c3c4a30
Sep 15 16:26:53.300890 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:53.300871    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37412cca01add92883383fcf18487e21.slice/crio-2ae18dd0a9ce0950d62ae70b70c3f8127cbece87a708689335058f84c74f10f1 WatchSource:0}: Error finding container 2ae18dd0a9ce0950d62ae70b70c3f8127cbece87a708689335058f84c74f10f1: Status 404 returned error can't find the container with id 2ae18dd0a9ce0950d62ae70b70c3f8127cbece87a708689335058f84c74f10f1
Sep 15 16:26:53.305841 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.305813    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:26:53.860707 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.860675    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:53.991039 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:53.991003    2566 apiserver.go:52] "Watching apiserver"
Sep 15 16:26:54.003826 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.003793    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 15 16:26:54.005004 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.004977    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-l5jb2","openshift-network-operator/iptables-alerter-ds64z","openshift-ovn-kubernetes/ovnkube-node-sc5tl","kube-system/global-pull-secret-syncer-zf92w","kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z","openshift-cluster-node-tuning-operator/tuned-mcl5w","openshift-image-registry/node-ca-xvx5n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal","openshift-multus/network-metrics-daemon-9t8nf","openshift-network-diagnostics/network-check-target-f9p97","kube-system/konnectivity-agent-rhpq7","openshift-dns/node-resolver-bpcsp","openshift-multus/multus-additional-cni-plugins-g4trk"]
Sep 15 16:26:54.007603 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.007566    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.009916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.009894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.010654 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.010636    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:26:54.010759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.010642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 15 16:26:54.010816 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.010771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jlxx6\""
Sep 15 16:26:54.012248 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.012228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.012359 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.012338    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:26:54.012985 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.012957    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 15 16:26:54.013440 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.013291    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 15 16:26:54.013440 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.013326    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5f6c\""
Sep 15 16:26:54.014430 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.014410    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 15 16:26:54.014671 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.014645    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.014975 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.014957    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 15 16:26:54.016082 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.016065    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 15 16:26:54.016365 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.016329    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 15 16:26:54.016365 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.016335    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 15 16:26:54.016519 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.016369    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wc26n\""
Sep 15 16:26:54.016519 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.016434    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 15 16:26:54.016519 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.016475    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 15 16:26:54.016980 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.016964    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 15 16:26:54.017447 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.017429    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kzqqp\""
Sep 15 16:26:54.017546 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.017507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.017980 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.017770    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 15 16:26:54.017980 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.017869    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 15 16:26:54.017980 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.017873    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 15 16:26:54.019691 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.019672    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 15 16:26:54.020057 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.020023    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.020152 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.020078    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wqb6z\""
Sep 15 16:26:54.020218 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.020183    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 15 16:26:54.020421 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.020402    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 15 16:26:54.022487 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.022455    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ckgm5\""
Sep 15 16:26:54.022587 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.022511    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 15 16:26:54.022719 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.022695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.022831 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.022770    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:26:54.022831 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.022825    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 15 16:26:54.022951 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.022840    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 15 16:26:54.025760 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.025740    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:54.025858 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.025815    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:26:54.028501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028479    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:26:54.028614 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.028562    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:26:54.028730 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-slash\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.028812 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-conf-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.028812 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-tuned\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.028812 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d95411-2800-4973-a18e-d4e8e03c3941-tmp\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.028812 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysctl-d\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.028812 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-systemd\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.029029 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028827    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-var-lib-kubelet\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.029029 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-netns\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.029029 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028859    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysconfig\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.029029 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/405e2425-ed77-4348-b777-be755c35de4d-host-slash\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.029029 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g7j5\" (UniqueName: \"kubernetes.io/projected/405e2425-ed77-4348-b777-be755c35de4d-kube-api-access-5g7j5\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.029029 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-kubelet\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.029029 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.028983    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-var-lib-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.029029 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-cni-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-cni-multus\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-ovnkube-script-lib\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-ovnkube-config\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfwf\" (UniqueName: \"kubernetes.io/projected/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-kube-api-access-wrfwf\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdcd828e-624a-4946-9a77-6e65028348a7-ovn-node-metrics-cert\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-cni-bin\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f64207aa-a010-4064-99ee-1ca2028b239e-host\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-run\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9pjw\" (UniqueName: \"kubernetes.io/projected/cdcd828e-624a-4946-9a77-6e65028348a7-kube-api-access-f9pjw\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-cnibin\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-cni-bin\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.029398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-hostroot\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-multus-certs\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-etc-kubernetes\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029479    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-host\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029508    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-system-cni-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029526    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-socket-dir-parent\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-sys\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029597    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-ovn\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-env-overrides\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-modprobe-d\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-kubernetes\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-lib-modules\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/405e2425-ed77-4348-b777-be755c35de4d-iptables-alerter-script\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-systemd\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-node-log\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-log-socket\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f64207aa-a010-4064-99ee-1ca2028b239e-serviceca\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.030042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.029985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-cni-netd\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2879h\" (UniqueName: \"kubernetes.io/projected/b5d95411-2800-4973-a18e-d4e8e03c3941-kube-api-access-2879h\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-k8s-cni-cncf-io\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-kubelet\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-systemd-units\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-cni-binary-copy\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030191    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysctl-conf\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-run-netns\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-etc-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-daemon-config\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.030799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bj8\" (UniqueName: \"kubernetes.io/projected/f64207aa-a010-4064-99ee-1ca2028b239e-kube-api-access-c9bj8\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.031295 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.030809    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 15 16:26:54.031295 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.031095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:26:54.033728 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.033710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.034902 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.034875    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 15 16:26:54.034902 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.034877    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 15 16:26:54.035268 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.035223    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mtkxk\""
Sep 15 16:26:54.037940 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.036040    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 15 16:26:54.037940 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.036728    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bbdlx\""
Sep 15 16:26:54.037940 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.037476    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 15 16:26:54.038589 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.038551    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.040795 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.040775    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pnnlg\""
Sep 15 16:26:54.040915 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.040886    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 15 16:26:54.041162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.041142    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 15 16:26:54.069041 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.069019    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ndn2t"
Sep 15 16:26:54.077865 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.077839    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ndn2t"
Sep 15 16:26:54.117830 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.117750    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 15 16:26:54.131332 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysctl-d\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.131480 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-systemd\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.131480 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131368    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-var-lib-kubelet\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.131480 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-netns\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.131480 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysconfig\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.131480 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4782e77-9ebd-4445-9a7d-59888a8916d4-konnectivity-ca\") pod \"konnectivity-agent-rhpq7\" (UID: \"c4782e77-9ebd-4445-9a7d-59888a8916d4\") " pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:26:54.131480 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-systemd\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-netns\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b80b8c-5e99-4d98-9193-0dd6056f00e5-tmp-dir\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131608    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/405e2425-ed77-4348-b777-be755c35de4d-host-slash\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-var-lib-kubelet\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysctl-d\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g7j5\" (UniqueName: \"kubernetes.io/projected/405e2425-ed77-4348-b777-be755c35de4d-kube-api-access-5g7j5\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131716    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/405e2425-ed77-4348-b777-be755c35de4d-host-slash\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.131740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysconfig\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-kubelet\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-kubelet\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-var-lib-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-cni-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.131936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-cni-multus\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-cni-multus\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-var-lib-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-system-cni-dir\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-cni-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-ovnkube-script-lib\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-ovnkube-config\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfwf\" (UniqueName: \"kubernetes.io/projected/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-kube-api-access-wrfwf\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdcd828e-624a-4946-9a77-6e65028348a7-ovn-node-metrics-cert\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-cni-bin\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f64207aa-a010-4064-99ee-1ca2028b239e-host\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-run\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-dbus\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9pjw\" (UniqueName: \"kubernetes.io/projected/cdcd828e-624a-4946-9a77-6e65028348a7-kube-api-access-f9pjw\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-cnibin\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-cni-bin\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-hostroot\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-multus-certs\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132523    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-etc-kubernetes\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.132573 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-host\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4782e77-9ebd-4445-9a7d-59888a8916d4-agent-certs\") pod \"konnectivity-agent-rhpq7\" (UID: \"c4782e77-9ebd-4445-9a7d-59888a8916d4\") " pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132616    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-system-cni-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132643    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-socket-dir-parent\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-sys\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55b80b8c-5e99-4d98-9193-0dd6056f00e5-hosts-file\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cnibin\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132837    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-sys-fs\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-ovn\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-env-overrides\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-ovnkube-config\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-socket-dir-parent\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-modprobe-d\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-kubernetes\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-sys\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-lib-modules\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.133450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.132989    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-hostroot\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hwff\" (UniqueName: \"kubernetes.io/projected/55b80b8c-5e99-4d98-9193-0dd6056f00e5-kube-api-access-4hwff\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-ovn\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-cni-bin\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133212    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-multus-certs\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-etc-kubernetes\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133309    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-host\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133397    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-modprobe-d\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-system-cni-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/405e2425-ed77-4348-b777-be755c35de4d-iptables-alerter-script\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-ovnkube-script-lib\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-systemd\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-kubernetes\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/cdcd828e-624a-4946-9a77-6e65028348a7-env-overrides\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-node-log\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-systemd\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-log-socket\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.134314 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133543    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f64207aa-a010-4064-99ee-1ca2028b239e-serviceca\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133559    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-lib-modules\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-node-log\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-cni-netd\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133654    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f64207aa-a010-4064-99ee-1ca2028b239e-host\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133694    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-log-socket\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2879h\" (UniqueName: \"kubernetes.io/projected/b5d95411-2800-4973-a18e-d4e8e03c3941-kube-api-access-2879h\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133736    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-cni-netd\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-run\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-run-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-k8s-cni-cncf-io\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-kubelet\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:54.135206 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133924    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-cnibin\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqlzl\" (UniqueName: \"kubernetes.io/projected/c3f47547-0e34-4911-a0e7-1f136f79d630-kube-api-access-wqlzl\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.133982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-systemd-units\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-cni-binary-copy\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134018    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/405e2425-ed77-4348-b777-be755c35de4d-iptables-alerter-script\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f64207aa-a010-4064-99ee-1ca2028b239e-serviceca\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134084    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysctl-conf\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-run-k8s-cni-cncf-io\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134173    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-cni-bin\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134213    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-host-var-lib-kubelet\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134253    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-systemd-units\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-kubelet-config\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-sysctl-conf\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134367    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshdt\" (UniqueName: \"kubernetes.io/projected/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-kube-api-access-pshdt\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134399    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-run-netns\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.136096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-etc-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-run-netns\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-etc-openvswitch\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-device-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-daemon-config\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bj8\" (UniqueName: \"kubernetes.io/projected/f64207aa-a010-4064-99ee-1ca2028b239e-kube-api-access-c9bj8\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-registration-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-slash\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-cni-binary-copy\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-conf-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-tuned\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cdcd828e-624a-4946-9a77-6e65028348a7-host-slash\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d95411-2800-4973-a18e-d4e8e03c3941-tmp\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-socket-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.137037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.134980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-conf-dir\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.137950 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.135023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9fkh\" (UniqueName: \"kubernetes.io/projected/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-kube-api-access-p9fkh\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.137950 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.135037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-multus-daemon-config\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.138126 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.138103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b5d95411-2800-4973-a18e-d4e8e03c3941-etc-tuned\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.138227 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.138208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b5d95411-2800-4973-a18e-d4e8e03c3941-tmp\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.138297 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.138278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/cdcd828e-624a-4946-9a77-6e65028348a7-ovn-node-metrics-cert\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.140818 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.140793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g7j5\" (UniqueName: \"kubernetes.io/projected/405e2425-ed77-4348-b777-be755c35de4d-kube-api-access-5g7j5\") pod \"iptables-alerter-ds64z\" (UID: \"405e2425-ed77-4348-b777-be755c35de4d\") " pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.146100 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.146072    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9pjw\" (UniqueName: \"kubernetes.io/projected/cdcd828e-624a-4946-9a77-6e65028348a7-kube-api-access-f9pjw\") pod \"ovnkube-node-sc5tl\" (UID: \"cdcd828e-624a-4946-9a77-6e65028348a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.146423 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.146405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfwf\" (UniqueName: \"kubernetes.io/projected/7a28728d-09b1-4b0a-97f4-bd78be16bb3c-kube-api-access-wrfwf\") pod \"multus-l5jb2\" (UID: \"7a28728d-09b1-4b0a-97f4-bd78be16bb3c\") " pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.147591 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.147572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bj8\" (UniqueName: \"kubernetes.io/projected/f64207aa-a010-4064-99ee-1ca2028b239e-kube-api-access-c9bj8\") pod \"node-ca-xvx5n\" (UID: \"f64207aa-a010-4064-99ee-1ca2028b239e\") " pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.147900 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.147874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2879h\" (UniqueName: \"kubernetes.io/projected/b5d95411-2800-4973-a18e-d4e8e03c3941-kube-api-access-2879h\") pod \"tuned-mcl5w\" (UID: \"b5d95411-2800-4973-a18e-d4e8e03c3941\") " pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.157844 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.157805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal" event={"ID":"37412cca01add92883383fcf18487e21","Type":"ContainerStarted","Data":"2ae18dd0a9ce0950d62ae70b70c3f8127cbece87a708689335058f84c74f10f1"}
Sep 15 16:26:54.159172 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.159135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal" event={"ID":"c252aca089984aa62b2a4fbb8eb23028","Type":"ContainerStarted","Data":"6bd7e11b49fc68a3555cc87fe1dd9afe42a1d9d9fff13bc3e1ac61e89c3c4a30"}
Sep 15 16:26:54.235837 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.235800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.235837 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.235835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-sys-fs\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.236044 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.235855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hwff\" (UniqueName: \"kubernetes.io/projected/55b80b8c-5e99-4d98-9193-0dd6056f00e5-kube-api-access-4hwff\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.236044 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.235918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-sys-fs\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.236044 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.235949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.236044 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.235983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.236044 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:54.236044 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqlzl\" (UniqueName: \"kubernetes.io/projected/c3f47547-0e34-4911-a0e7-1f136f79d630-kube-api-access-wqlzl\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-kubelet-config\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshdt\" (UniqueName: \"kubernetes.io/projected/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-kube-api-access-pshdt\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-device-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.236172    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-registration-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.236289    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs podName:c3f47547-0e34-4911-a0e7-1f136f79d630 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:54.736248793 +0000 UTC m=+3.104073999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs") pod "network-metrics-daemon-9t8nf" (UID: "c3f47547-0e34-4911-a0e7-1f136f79d630") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236301    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.236353 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.236318    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-kubelet-config\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.236495    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret podName:8e9dce9b-8568-498c-a7f8-cfe8d55410c1 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:54.736474751 +0000 UTC m=+3.104299930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret") pod "global-pull-secret-syncer-zf92w" (UID: "8e9dce9b-8568-498c-a7f8-cfe8d55410c1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-socket-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-registration-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-device-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9fkh\" (UniqueName: \"kubernetes.io/projected/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-kube-api-access-p9fkh\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4782e77-9ebd-4445-9a7d-59888a8916d4-konnectivity-ca\") pod \"konnectivity-agent-rhpq7\" (UID: \"c4782e77-9ebd-4445-9a7d-59888a8916d4\") " pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b80b8c-5e99-4d98-9193-0dd6056f00e5-tmp-dir\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-socket-dir\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-system-cni-dir\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-dbus\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4782e77-9ebd-4445-9a7d-59888a8916d4-agent-certs\") pod \"konnectivity-agent-rhpq7\" (UID: \"c4782e77-9ebd-4445-9a7d-59888a8916d4\") " pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55b80b8c-5e99-4d98-9193-0dd6056f00e5-hosts-file\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.237024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cnibin\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236901    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-dbus\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.236913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cnibin\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.237012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b80b8c-5e99-4d98-9193-0dd6056f00e5-tmp-dir\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.237036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-etc-selinux\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.237083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-system-cni-dir\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.237097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/55b80b8c-5e99-4d98-9193-0dd6056f00e5-hosts-file\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.237102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.237218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c4782e77-9ebd-4445-9a7d-59888a8916d4-konnectivity-ca\") pod \"konnectivity-agent-rhpq7\" (UID: \"c4782e77-9ebd-4445-9a7d-59888a8916d4\") " pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:26:54.237662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.237421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-cni-binary-copy\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.239256 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.239235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c4782e77-9ebd-4445-9a7d-59888a8916d4-agent-certs\") pod \"konnectivity-agent-rhpq7\" (UID: \"c4782e77-9ebd-4445-9a7d-59888a8916d4\") " pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:26:54.245686 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.245638    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:54.245686 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.245683    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:54.245863 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.245699    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqvn4 for pod openshift-network-diagnostics/network-check-target-f9p97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:54.245863 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.245771    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4 podName:c9072b15-5772-442d-9666-6be71bffbe35 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:54.745751601 +0000 UTC m=+3.113576793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zqvn4" (UniqueName: "kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4") pod "network-check-target-f9p97" (UID: "c9072b15-5772-442d-9666-6be71bffbe35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:54.247493 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.247469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hwff\" (UniqueName: \"kubernetes.io/projected/55b80b8c-5e99-4d98-9193-0dd6056f00e5-kube-api-access-4hwff\") pod \"node-resolver-bpcsp\" (UID: \"55b80b8c-5e99-4d98-9193-0dd6056f00e5\") " pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.247944 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.247919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqlzl\" (UniqueName: \"kubernetes.io/projected/c3f47547-0e34-4911-a0e7-1f136f79d630-kube-api-access-wqlzl\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:54.248244 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.248219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9fkh\" (UniqueName: \"kubernetes.io/projected/4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f-kube-api-access-p9fkh\") pod \"aws-ebs-csi-driver-node-92r5z\" (UID: \"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.248896 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.248878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshdt\" (UniqueName: \"kubernetes.io/projected/35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1-kube-api-access-pshdt\") pod \"multus-additional-cni-plugins-g4trk\" (UID: \"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1\") " pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.320849 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.320814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mcl5w"
Sep 15 16:26:54.328572 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.328549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ds64z"
Sep 15 16:26:54.339882 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.339862    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:26:54.346801 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.346782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l5jb2"
Sep 15 16:26:54.354535 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.354514    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-xvx5n"
Sep 15 16:26:54.361357 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.361337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z"
Sep 15 16:26:54.369222 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.369168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:26:54.376821 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.376796    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bpcsp"
Sep 15 16:26:54.383621 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.383600    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g4trk"
Sep 15 16:26:54.571263 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.571233    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a28728d_09b1_4b0a_97f4_bd78be16bb3c.slice/crio-6ca567156a721983b80fec8a5e1cbb21112db578ce0671749c1a3ec8865e08e1 WatchSource:0}: Error finding container 6ca567156a721983b80fec8a5e1cbb21112db578ce0671749c1a3ec8865e08e1: Status 404 returned error can't find the container with id 6ca567156a721983b80fec8a5e1cbb21112db578ce0671749c1a3ec8865e08e1
Sep 15 16:26:54.572398 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.572372    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d95411_2800_4973_a18e_d4e8e03c3941.slice/crio-efb3ac09de871500c6f703961a01bf81c59ed4c76df90186fa850c79bcf07ece WatchSource:0}: Error finding container efb3ac09de871500c6f703961a01bf81c59ed4c76df90186fa850c79bcf07ece: Status 404 returned error can't find the container with id efb3ac09de871500c6f703961a01bf81c59ed4c76df90186fa850c79bcf07ece
Sep 15 16:26:54.573197 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.573173    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405e2425_ed77_4348_b777_be755c35de4d.slice/crio-015005d3e44d150c3b7e5daee7f5826e633921594f2e09a3bde8b37cff95d28b WatchSource:0}: Error finding container 015005d3e44d150c3b7e5daee7f5826e633921594f2e09a3bde8b37cff95d28b: Status 404 returned error can't find the container with id 015005d3e44d150c3b7e5daee7f5826e633921594f2e09a3bde8b37cff95d28b
Sep 15 16:26:54.576594 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.576566    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd4fb83_7e9f_444e_ab4f_c265edc0bc5f.slice/crio-465fafed4596fd0e7e848f379bc5716ebb6e3776f3dca5226e3f8f0f390923ed WatchSource:0}: Error finding container 465fafed4596fd0e7e848f379bc5716ebb6e3776f3dca5226e3f8f0f390923ed: Status 404 returned error can't find the container with id 465fafed4596fd0e7e848f379bc5716ebb6e3776f3dca5226e3f8f0f390923ed
Sep 15 16:26:54.577330 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.577287    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b80b8c_5e99_4d98_9193_0dd6056f00e5.slice/crio-253f62ac80f49792852e389560e92783f4a1118ae440abcd8baadee17e8a6fda WatchSource:0}: Error finding container 253f62ac80f49792852e389560e92783f4a1118ae440abcd8baadee17e8a6fda: Status 404 returned error can't find the container with id 253f62ac80f49792852e389560e92783f4a1118ae440abcd8baadee17e8a6fda
Sep 15 16:26:54.578791 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.578692    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcd828e_624a_4946_9a77_6e65028348a7.slice/crio-5e09e1a4695a6721e48500f48d700583c8f808f43d49cab8683d93d9f1408493 WatchSource:0}: Error finding container 5e09e1a4695a6721e48500f48d700583c8f808f43d49cab8683d93d9f1408493: Status 404 returned error can't find the container with id 5e09e1a4695a6721e48500f48d700583c8f808f43d49cab8683d93d9f1408493
Sep 15 16:26:54.579790 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.579761    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64207aa_a010_4064_99ee_1ca2028b239e.slice/crio-09b06370c238995771829b99c2a89a23ac35ee1ba4712844b1b2f7d72c683af9 WatchSource:0}: Error finding container 09b06370c238995771829b99c2a89a23ac35ee1ba4712844b1b2f7d72c683af9: Status 404 returned error can't find the container with id 09b06370c238995771829b99c2a89a23ac35ee1ba4712844b1b2f7d72c683af9
Sep 15 16:26:54.580864 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.580836    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35c5b9ae_2d18_41e2_afdd_a4b6d25b18c1.slice/crio-2641218eef932a4d2000a1a25b50a3f604ec2d485dc0d1917c4e15fc36853c13 WatchSource:0}: Error finding container 2641218eef932a4d2000a1a25b50a3f604ec2d485dc0d1917c4e15fc36853c13: Status 404 returned error can't find the container with id 2641218eef932a4d2000a1a25b50a3f604ec2d485dc0d1917c4e15fc36853c13
Sep 15 16:26:54.581898 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:26:54.581878    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4782e77_9ebd_4445_9a7d_59888a8916d4.slice/crio-9c2219dcaf3a3980a16e3fd663dfc36e008846b8472d7d8ad81cfd453715e008 WatchSource:0}: Error finding container 9c2219dcaf3a3980a16e3fd663dfc36e008846b8472d7d8ad81cfd453715e008: Status 404 returned error can't find the container with id 9c2219dcaf3a3980a16e3fd663dfc36e008846b8472d7d8ad81cfd453715e008
Sep 15 16:26:54.740975 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.740929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:54.740975 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.740980    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:54.741238 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.741089    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:54.741238 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.741092    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:54.741238 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.741149    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret podName:8e9dce9b-8568-498c-a7f8-cfe8d55410c1 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:55.741129326 +0000 UTC m=+4.108954521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret") pod "global-pull-secret-syncer-zf92w" (UID: "8e9dce9b-8568-498c-a7f8-cfe8d55410c1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:54.741238 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.741169    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs podName:c3f47547-0e34-4911-a0e7-1f136f79d630 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:55.741158791 +0000 UTC m=+4.108983969 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs") pod "network-metrics-daemon-9t8nf" (UID: "c3f47547-0e34-4911-a0e7-1f136f79d630") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:54.841841 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:54.841815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:26:54.841986 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.841971    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:54.842026 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.841993    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:54.842026 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.842003    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqvn4 for pod openshift-network-diagnostics/network-check-target-f9p97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:54.842106 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:54.842052    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4 podName:c9072b15-5772-442d-9666-6be71bffbe35 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:55.842037963 +0000 UTC m=+4.209863156 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqvn4" (UniqueName: "kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4") pod "network-check-target-f9p97" (UID: "c9072b15-5772-442d-9666-6be71bffbe35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:55.082126 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.082018    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:21:54 +0000 UTC" deadline="2028-05-08 04:51:00.148400396 +0000 UTC"
Sep 15 16:26:55.082126 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.082064    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14412h24m5.066341686s"
Sep 15 16:26:55.169310 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.169204    2566 generic.go:356] "Generic (PLEG): container finished" podID="37412cca01add92883383fcf18487e21" containerID="3e9e9f449eb168d14e033a3dc477d0119a616d46d38a2ea6716277dd59dccd74" exitCode=0
Sep 15 16:26:55.169310 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.169297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal" event={"ID":"37412cca01add92883383fcf18487e21","Type":"ContainerDied","Data":"3e9e9f449eb168d14e033a3dc477d0119a616d46d38a2ea6716277dd59dccd74"}
Sep 15 16:26:55.173806 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.173734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerStarted","Data":"2641218eef932a4d2000a1a25b50a3f604ec2d485dc0d1917c4e15fc36853c13"}
Sep 15 16:26:55.178422 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.178332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xvx5n" event={"ID":"f64207aa-a010-4064-99ee-1ca2028b239e","Type":"ContainerStarted","Data":"09b06370c238995771829b99c2a89a23ac35ee1ba4712844b1b2f7d72c683af9"}
Sep 15 16:26:55.185578 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.185464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"5e09e1a4695a6721e48500f48d700583c8f808f43d49cab8683d93d9f1408493"}
Sep 15 16:26:55.188360 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.188309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bpcsp" event={"ID":"55b80b8c-5e99-4d98-9193-0dd6056f00e5","Type":"ContainerStarted","Data":"253f62ac80f49792852e389560e92783f4a1118ae440abcd8baadee17e8a6fda"}
Sep 15 16:26:55.192609 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.192578    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z" event={"ID":"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f","Type":"ContainerStarted","Data":"465fafed4596fd0e7e848f379bc5716ebb6e3776f3dca5226e3f8f0f390923ed"}
Sep 15 16:26:55.196757 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.196708    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mcl5w" event={"ID":"b5d95411-2800-4973-a18e-d4e8e03c3941","Type":"ContainerStarted","Data":"efb3ac09de871500c6f703961a01bf81c59ed4c76df90186fa850c79bcf07ece"}
Sep 15 16:26:55.198813 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.198734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rhpq7" event={"ID":"c4782e77-9ebd-4445-9a7d-59888a8916d4","Type":"ContainerStarted","Data":"9c2219dcaf3a3980a16e3fd663dfc36e008846b8472d7d8ad81cfd453715e008"}
Sep 15 16:26:55.213999 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.213856    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ds64z" event={"ID":"405e2425-ed77-4348-b777-be755c35de4d","Type":"ContainerStarted","Data":"015005d3e44d150c3b7e5daee7f5826e633921594f2e09a3bde8b37cff95d28b"}
Sep 15 16:26:55.216768 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.216735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5jb2" event={"ID":"7a28728d-09b1-4b0a-97f4-bd78be16bb3c","Type":"ContainerStarted","Data":"6ca567156a721983b80fec8a5e1cbb21112db578ce0671749c1a3ec8865e08e1"}
Sep 15 16:26:55.353966 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.353936    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 15 16:26:55.754511 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.753717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:55.754511 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.753794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:55.754511 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:55.753966    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:55.754511 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:55.754033    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret podName:8e9dce9b-8568-498c-a7f8-cfe8d55410c1 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:57.754013147 +0000 UTC m=+6.121838341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret") pod "global-pull-secret-syncer-zf92w" (UID: "8e9dce9b-8568-498c-a7f8-cfe8d55410c1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:55.754511 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:55.754428    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:55.754511 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:55.754476    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs podName:c3f47547-0e34-4911-a0e7-1f136f79d630 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:57.754460633 +0000 UTC m=+6.122285827 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs") pod "network-metrics-daemon-9t8nf" (UID: "c3f47547-0e34-4911-a0e7-1f136f79d630") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:55.855170 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:55.855129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:26:55.855403 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:55.855343    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:55.855403 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:55.855368    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:55.855403 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:55.855382    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqvn4 for pod openshift-network-diagnostics/network-check-target-f9p97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:55.855601 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:55.855448    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4 podName:c9072b15-5772-442d-9666-6be71bffbe35 nodeName:}" failed. No retries permitted until 2026-09-15 16:26:57.855427072 +0000 UTC m=+6.223252272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqvn4" (UniqueName: "kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4") pod "network-check-target-f9p97" (UID: "c9072b15-5772-442d-9666-6be71bffbe35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:56.083319 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:56.082985    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-14 16:21:54 +0000 UTC" deadline="2028-03-06 08:46:39.666788743 +0000 UTC"
Sep 15 16:26:56.083319 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:56.083268    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12904h19m43.58352762s"
Sep 15 16:26:56.153912 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:56.153814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:56.154083 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:56.153975    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:26:56.154454 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:56.154398    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:26:56.154557 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:56.154502    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:26:56.154639 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:56.154577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:56.154692 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:56.154674    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:26:56.241154 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:56.241107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal" event={"ID":"37412cca01add92883383fcf18487e21","Type":"ContainerStarted","Data":"b1646d53d5d10f32bd292ac93e3d9b2dfd5cc1072777d6fcb6bbfeaf3c942451"}
Sep 15 16:26:57.771575 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:57.771533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:57.772140 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:57.771627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:57.772140 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:57.771799    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:57.772140 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:57.771869    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret podName:8e9dce9b-8568-498c-a7f8-cfe8d55410c1 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:01.771850916 +0000 UTC m=+10.139676111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret") pod "global-pull-secret-syncer-zf92w" (UID: "8e9dce9b-8568-498c-a7f8-cfe8d55410c1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:26:57.772314 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:57.772272    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:57.772363 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:57.772322    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs podName:c3f47547-0e34-4911-a0e7-1f136f79d630 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:01.772307305 +0000 UTC m=+10.140132500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs") pod "network-metrics-daemon-9t8nf" (UID: "c3f47547-0e34-4911-a0e7-1f136f79d630") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:26:57.872465 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:57.872424    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:26:57.872648 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:57.872616    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:26:57.872648 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:57.872643    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:26:57.872767 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:57.872659    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqvn4 for pod openshift-network-diagnostics/network-check-target-f9p97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:57.872767 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:57.872720    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4 podName:c9072b15-5772-442d-9666-6be71bffbe35 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:01.872701084 +0000 UTC m=+10.240526265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqvn4" (UniqueName: "kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4") pod "network-check-target-f9p97" (UID: "c9072b15-5772-442d-9666-6be71bffbe35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:26:58.156347 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:58.155516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:26:58.156347 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:58.155670    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:26:58.156347 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:58.156067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:26:58.156347 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:58.156150    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:26:58.156347 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:26:58.156191    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:26:58.156347 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:26:58.156255    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:00.153028 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:00.152990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:00.153473 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:00.153180    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:00.153473 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:00.153349    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:00.153650 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:00.153491    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:00.153650 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:00.153528    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:00.153650 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:00.153612    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:01.808382 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:01.808347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:01.808824 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:01.808413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:01.808824 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:01.808510    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:01.808824 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:01.808546    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:01.808824 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:01.808601    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs podName:c3f47547-0e34-4911-a0e7-1f136f79d630 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:09.808565816 +0000 UTC m=+18.176391007 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs") pod "network-metrics-daemon-9t8nf" (UID: "c3f47547-0e34-4911-a0e7-1f136f79d630") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:01.808824 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:01.808621    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret podName:8e9dce9b-8568-498c-a7f8-cfe8d55410c1 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:09.808611512 +0000 UTC m=+18.176436694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret") pod "global-pull-secret-syncer-zf92w" (UID: "8e9dce9b-8568-498c-a7f8-cfe8d55410c1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:01.910236 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:01.909557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:01.910236 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:01.909801    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:01.910236 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:01.909824    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:01.910236 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:01.909838    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqvn4 for pod openshift-network-diagnostics/network-check-target-f9p97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:01.910236 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:01.909896    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4 podName:c9072b15-5772-442d-9666-6be71bffbe35 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:09.909878485 +0000 UTC m=+18.277703676 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqvn4" (UniqueName: "kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4") pod "network-check-target-f9p97" (UID: "c9072b15-5772-442d-9666-6be71bffbe35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:02.154564 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:02.154479    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:02.154772 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:02.154602    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:02.154772 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:02.154614    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:02.154772 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:02.154762    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:02.154956 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:02.154797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:02.154956 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:02.154874    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:04.153625 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:04.153553    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:04.153625 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:04.153600    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:04.153625 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:04.153608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:04.154207 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:04.153714    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:04.154477 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:04.154379    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:04.154618 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:04.154546    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:06.153410 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:06.153376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:06.153410 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:06.153388    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:06.153931 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:06.153387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:06.153931 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:06.153496    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:06.153931 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:06.153620    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:06.153931 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:06.153700    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:08.153830 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:08.153787    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:08.154272 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:08.153788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:08.154272 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:08.153929    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:08.154272 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:08.153788    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:08.154272 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:08.154015    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:08.154272 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:08.154127    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:09.867298 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:09.867262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:09.867655 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:09.867314    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:09.867655 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:09.867399    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:09.867655 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:09.867418    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:09.867655 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:09.867455    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret podName:8e9dce9b-8568-498c-a7f8-cfe8d55410c1 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.86743944 +0000 UTC m=+34.235264637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret") pod "global-pull-secret-syncer-zf92w" (UID: "8e9dce9b-8568-498c-a7f8-cfe8d55410c1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:09.867655 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:09.867470    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs podName:c3f47547-0e34-4911-a0e7-1f136f79d630 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.867463282 +0000 UTC m=+34.235288459 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs") pod "network-metrics-daemon-9t8nf" (UID: "c3f47547-0e34-4911-a0e7-1f136f79d630") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:09.968046 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:09.968009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:09.968199 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:09.968163    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:09.968199 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:09.968185    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:09.968199 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:09.968195    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqvn4 for pod openshift-network-diagnostics/network-check-target-f9p97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:09.968315 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:09.968249    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4 podName:c9072b15-5772-442d-9666-6be71bffbe35 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:25.96823527 +0000 UTC m=+34.336060446 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqvn4" (UniqueName: "kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4") pod "network-check-target-f9p97" (UID: "c9072b15-5772-442d-9666-6be71bffbe35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:10.153888 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:10.153795    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:10.154055 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:10.153799    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:10.154055 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:10.153800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:10.154055 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:10.153939    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:10.154211 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:10.154067    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:10.154211 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:10.154163    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:12.154297 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:12.154255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:12.154781 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:12.154369    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:12.154781 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:12.154455    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:12.154781 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:12.154597    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:12.154781 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:12.154599    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:12.154781 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:12.154685    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:14.154062 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.153759    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:14.154965 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.153759    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:14.154965 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:14.154145    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:14.154965 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.153760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:14.154965 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:14.154200    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:14.154965 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:14.154296    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:14.278301 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.278274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-xvx5n" event={"ID":"f64207aa-a010-4064-99ee-1ca2028b239e","Type":"ContainerStarted","Data":"6d849abadddd428ef8cc0494ae2f2f6d720cc25a1ce1cbf73b5f063a7854d84d"}
Sep 15 16:27:14.280876 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.280854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"d4bdf8021da6730ee28ddac3287a54e7b4356d469ee319a9900f27317a7b76eb"}
Sep 15 16:27:14.280973 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.280916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"40ddf2a512504d795cd7d7e80285162523d519b2f90b046e3e5d4e2025367745"}
Sep 15 16:27:14.280973 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.280955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"3e60c1eab40b7248327b2f39cedfe7eb63b24e1ab12b253d054fcf3545b440f9"}
Sep 15 16:27:14.281077 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.280997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"711e403e53250484b733bd8338e0bb8b8061c7f796f99b0f7fd82d390650179f"}
Sep 15 16:27:14.282303 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.282276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bpcsp" event={"ID":"55b80b8c-5e99-4d98-9193-0dd6056f00e5","Type":"ContainerStarted","Data":"6e6622b07b798e76cde52a68cff54a55dddccb6996814c9fc2d9803150f9de30"}
Sep 15 16:27:14.283731 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.283711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z" event={"ID":"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f","Type":"ContainerStarted","Data":"680057421bcf1f54a857c7a7934414183ff9ace7113c70b5fe5853a6cc36e19d"}
Sep 15 16:27:14.285031 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.285013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mcl5w" event={"ID":"b5d95411-2800-4973-a18e-d4e8e03c3941","Type":"ContainerStarted","Data":"38aa469fefd30513bf4e9df1689f5209a184e97baadb867d466d393adaca642b"}
Sep 15 16:27:14.286359 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.286339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rhpq7" event={"ID":"c4782e77-9ebd-4445-9a7d-59888a8916d4","Type":"ContainerStarted","Data":"b8ec719b4c347b6741904147d0ca3062dca7e9077874e3d1c26e8521b7d6b259"}
Sep 15 16:27:14.287717 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.287693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l5jb2" event={"ID":"7a28728d-09b1-4b0a-97f4-bd78be16bb3c","Type":"ContainerStarted","Data":"754f455bb0282a59bdc3c622679ea72fa90c7a0dc2017f65cada492fe8735bfa"}
Sep 15 16:27:14.288936 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.288907    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal" event={"ID":"c252aca089984aa62b2a4fbb8eb23028","Type":"ContainerStarted","Data":"6af0d7007788129621bc22b3611ddd563886a95692b79e40c3c592b4441a0bb4"}
Sep 15 16:27:14.290200 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.290180    2566 generic.go:356] "Generic (PLEG): container finished" podID="35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1" containerID="f65f48ed4cac13ddf7a464f677f91edaba4ba99f723d862a1b7dba34baf2988a" exitCode=0
Sep 15 16:27:14.290300 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.290206    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerDied","Data":"f65f48ed4cac13ddf7a464f677f91edaba4ba99f723d862a1b7dba34baf2988a"}
Sep 15 16:27:14.292561 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.292518    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-xvx5n" podStartSLOduration=3.195982763 podStartE2EDuration="22.292504971s" totalImagesPullingTime="19.096522208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:14.292403625 +0000 UTC m=+22.660228824" watchObservedRunningTime="2026-09-15 16:27:14.292504971 +0000 UTC m=+22.660330183"
Sep 15 16:27:14.292663 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.292612    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-85.ec2.internal" podStartSLOduration=21.292606505 podStartE2EDuration="21.292606505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:26:56.262962366 +0000 UTC m=+4.630787570" watchObservedRunningTime="2026-09-15 16:27:14.292606505 +0000 UTC m=+22.660431718"
Sep 15 16:27:14.311904 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.311866    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mcl5w" podStartSLOduration=3.12297664 podStartE2EDuration="22.311854354s" totalImagesPullingTime="19.188877714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:14.311788815 +0000 UTC m=+22.679614025" watchObservedRunningTime="2026-09-15 16:27:14.311854354 +0000 UTC m=+22.679679552"
Sep 15 16:27:14.331042 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.331006    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l5jb2" podStartSLOduration=3.077489359 podStartE2EDuration="22.330993866s" totalImagesPullingTime="19.253504507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:14.330764093 +0000 UTC m=+22.698589292" watchObservedRunningTime="2026-09-15 16:27:14.330993866 +0000 UTC m=+22.698819065"
Sep 15 16:27:14.367113 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.367076    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rhpq7" podStartSLOduration=6.15857703 podStartE2EDuration="21.367062471s" totalImagesPullingTime="15.208485441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:14.347835684 +0000 UTC m=+22.715660883" watchObservedRunningTime="2026-09-15 16:27:14.367062471 +0000 UTC m=+22.734887669"
Sep 15 16:27:14.385546 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.385491    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-85.ec2.internal" podStartSLOduration=21.385472003 podStartE2EDuration="21.385472003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:14.366853228 +0000 UTC m=+22.734678452" watchObservedRunningTime="2026-09-15 16:27:14.385472003 +0000 UTC m=+22.753297207"
Sep 15 16:27:14.386335 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:14.386288    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-bpcsp" podStartSLOduration=2.263493033 podStartE2EDuration="21.386274859s" totalImagesPullingTime="19.122781826s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:14.385748454 +0000 UTC m=+22.753573658" watchObservedRunningTime="2026-09-15 16:27:14.386274859 +0000 UTC m=+22.754100061"
Sep 15 16:27:15.234018 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:15.233950    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 15 16:27:15.294768 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:15.294733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z" event={"ID":"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f","Type":"ContainerStarted","Data":"7141fd38bbe5e0cb3dd7e468c99c3b97d6456a3b3e68fcb4340e5d84bb1f7741"}
Sep 15 16:27:15.296532 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:15.296500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ds64z" event={"ID":"405e2425-ed77-4348-b777-be755c35de4d","Type":"ContainerStarted","Data":"8ae41942c8f0b3d298dfdfdd883712be18a2399ed5ae1434a2eb873c70e81d60"}
Sep 15 16:27:15.300193 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:15.300165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"861546e9b715d662c929998d69139c4bc02262874fd766dd96a8f601fc8dbc41"}
Sep 15 16:27:15.300317 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:15.300244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"913c2509f12021f75ab883c0e564eb5e259e78d3a9fd39a11eb0e0da573dcfc8"}
Sep 15 16:27:15.309969 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:15.309924    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ds64z" podStartSLOduration=4.121985006 podStartE2EDuration="23.309906022s" totalImagesPullingTime="19.187921016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:15.309785026 +0000 UTC m=+23.677610226" watchObservedRunningTime="2026-09-15 16:27:15.309906022 +0000 UTC m=+23.677731223"
Sep 15 16:27:16.153728 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:16.153678    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:16.153728 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:16.153704    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:16.154063 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:16.153709    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:16.154063 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:16.153814    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:16.154063 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:16.153889    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:16.154063 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:16.153977    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:16.222417 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:16.222313    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-15T16:27:15.233972173Z","UUID":"e0f5f947-d792-4bda-a992-00a8d15bc18a","Handler":null,"Name":"","Endpoint":""}
Sep 15 16:27:16.223968 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:16.223944    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 15 16:27:16.224095 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:16.223979    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 15 16:27:16.512065 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:16.512031    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:27:16.512742 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:16.512719    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:27:17.309343 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:17.309248    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"9b6b97894e2c5363dc40bf6bc1d9129b12e6e97823b40753e363013461d1b42a"}
Sep 15 16:27:17.311972 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:17.311944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z" event={"ID":"4cd4fb83-7e9f-444e-ab4f-c265edc0bc5f","Type":"ContainerStarted","Data":"f005673873888a20069e72e3fa24dbdb65f4613343b8d7dad22a47b484b4a8c6"}
Sep 15 16:27:17.312200 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:17.312184    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:27:17.312637 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:17.312617    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rhpq7"
Sep 15 16:27:17.331542 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:17.331503    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-92r5z" podStartSLOduration=3.381733687 podStartE2EDuration="25.331490873s" totalImagesPullingTime="21.949757186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:17.33097367 +0000 UTC m=+25.698798869" watchObservedRunningTime="2026-09-15 16:27:17.331490873 +0000 UTC m=+25.699316072"
Sep 15 16:27:18.153976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:18.153747    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:18.154455 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:18.153777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:18.154455 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:18.154091    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:18.154455 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:18.153851    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:18.154455 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:18.154151    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:18.154455 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:18.154231    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:19.318968 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:19.318930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerStarted","Data":"01db84087f77d32cd3240521f30e7e5e611f46b03d534320fbef58fe7e0dce74"}
Sep 15 16:27:19.324237 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:19.324206    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" event={"ID":"cdcd828e-624a-4946-9a77-6e65028348a7","Type":"ContainerStarted","Data":"1ba40f125df212d89c73133d59e3acca4fbe4698e20b0498df8cb7b3b7ef9ef1"}
Sep 15 16:27:19.324500 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:19.324470    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:27:19.339943 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:19.339916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:27:19.376648 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:19.376435    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl" podStartSLOduration=8.15499937 podStartE2EDuration="27.376418184s" totalImagesPullingTime="19.221418814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:19.374889799 +0000 UTC m=+27.742714997" watchObservedRunningTime="2026-09-15 16:27:19.376418184 +0000 UTC m=+27.744243383"
Sep 15 16:27:20.153199 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:20.153164    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:20.153358 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:20.153173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:20.153358 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:20.153283    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:20.153466 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:20.153387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:20.153466 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:20.153459    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:20.153540 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:20.153380    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:20.328709 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:20.328672    2566 generic.go:356] "Generic (PLEG): container finished" podID="35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1" containerID="01db84087f77d32cd3240521f30e7e5e611f46b03d534320fbef58fe7e0dce74" exitCode=0
Sep 15 16:27:20.329083 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:20.328761    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerDied","Data":"01db84087f77d32cd3240521f30e7e5e611f46b03d534320fbef58fe7e0dce74"}
Sep 15 16:27:20.329083 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:20.328966    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 16:27:20.332212 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:20.332096    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:27:20.348174 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:20.348148    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:27:21.207144 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.207116    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9t8nf"]
Sep 15 16:27:21.207260 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.207245    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:21.207382 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:21.207357    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:21.210443 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.210417    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f9p97"]
Sep 15 16:27:21.210588 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.210556    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:21.210686 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:21.210664    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:21.224630 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.224601    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zf92w"]
Sep 15 16:27:21.224743 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.224720    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:21.224842 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:21.224812    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:21.332995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.332962    2566 generic.go:356] "Generic (PLEG): container finished" podID="35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1" containerID="6ac33269bd74009a878121498ac5d9f3da90732a3a70977c45589aa8422471eb" exitCode=0
Sep 15 16:27:21.333427 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.333040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerDied","Data":"6ac33269bd74009a878121498ac5d9f3da90732a3a70977c45589aa8422471eb"}
Sep 15 16:27:21.333427 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:21.333102    2566 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 15 16:27:22.073342 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:22.073314    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:27:23.153473 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:23.153390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:23.153914 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:23.153486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:23.153914 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:23.153491    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:23.153914 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:23.153533    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:23.153914 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:23.153649    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:23.153914 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:23.153749    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:23.340759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:23.340726    2566 generic.go:356] "Generic (PLEG): container finished" podID="35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1" containerID="f2efd2e8facce77f301fb7ed6deb58124f9026c8a21e9a886c484d4180c7cb58" exitCode=0
Sep 15 16:27:23.340861 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:23.340814    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerDied","Data":"f2efd2e8facce77f301fb7ed6deb58124f9026c8a21e9a886c484d4180c7cb58"}
Sep 15 16:27:25.153561 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:25.153370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:25.153964 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:25.153410    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:25.153964 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.153652    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-f9p97" podUID="c9072b15-5772-442d-9666-6be71bffbe35"
Sep 15 16:27:25.153964 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.153729    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zf92w" podUID="8e9dce9b-8568-498c-a7f8-cfe8d55410c1"
Sep 15 16:27:25.153964 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:25.153414    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:25.153964 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.153848    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9t8nf" podUID="c3f47547-0e34-4911-a0e7-1f136f79d630"
Sep 15 16:27:25.884453 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:25.884416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:25.884670 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:25.884483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:25.884670 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.884606    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:25.884670 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.884619    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:25.884815 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.884676    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs podName:c3f47547-0e34-4911-a0e7-1f136f79d630 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:57.884661633 +0000 UTC m=+66.252486824 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs") pod "network-metrics-daemon-9t8nf" (UID: "c3f47547-0e34-4911-a0e7-1f136f79d630") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 15 16:27:25.884815 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.884694    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret podName:8e9dce9b-8568-498c-a7f8-cfe8d55410c1 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:57.884686082 +0000 UTC m=+66.252511288 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret") pod "global-pull-secret-syncer-zf92w" (UID: "8e9dce9b-8568-498c-a7f8-cfe8d55410c1") : object "kube-system"/"original-pull-secret" not registered
Sep 15 16:27:25.985808 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:25.985774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:25.986029 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.985960    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 15 16:27:25.986029 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.985990    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 15 16:27:25.986029 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.986005    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zqvn4 for pod openshift-network-diagnostics/network-check-target-f9p97: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:25.986170 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:25.986070    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4 podName:c9072b15-5772-442d-9666-6be71bffbe35 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:57.986051372 +0000 UTC m=+66.353876553 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zqvn4" (UniqueName: "kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4") pod "network-check-target-f9p97" (UID: "c9072b15-5772-442d-9666-6be71bffbe35") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 15 16:27:26.382433 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.382400    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-85.ec2.internal" event="NodeReady"
Sep 15 16:27:26.382956 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.382548    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 15 16:27:26.468010 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.467979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d87d77784-67zfs"]
Sep 15 16:27:26.494587 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.494538    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp"]
Sep 15 16:27:26.494775 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.494691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.502176 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.502139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-whzsv\""
Sep 15 16:27:26.502399 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.502218    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 15 16:27:26.502833 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.502812    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 15 16:27:26.503783 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.503759    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 15 16:27:26.508285 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.508252    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d87d77784-67zfs"]
Sep 15 16:27:26.508397 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.508314    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l9b4d"]
Sep 15 16:27:26.508499 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.508475    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp"
Sep 15 16:27:26.509295 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.509275    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 15 16:27:26.522827 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.522800    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"]
Sep 15 16:27:26.522951 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.522877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.528476 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.528454    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.528703 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.528681    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-srrvk\""
Sep 15 16:27:26.531773 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.531752    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.538048 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.538000    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-nhpth"]
Sep 15 16:27:26.538797 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.538772    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:26.545037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.545016    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mzxqz\""
Sep 15 16:27:26.546204 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.546183    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 15 16:27:26.546467 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.546451    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 15 16:27:26.548131 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.548082    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 15 16:27:26.548434 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.548416    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6cgc5\""
Sep 15 16:27:26.548802 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.548786    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.549118 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.549104    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.554121 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.554102    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"]
Sep 15 16:27:26.554256 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.554234    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.557376 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.557358    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.558577 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.558541    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 15 16:27:26.559503 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.559454    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 15 16:27:26.559648 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.559472    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6xrz5\""
Sep 15 16:27:26.559727 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.559492    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.566067 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.566046    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 15 16:27:26.574955 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.574927    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"]
Sep 15 16:27:26.575074 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.575055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.577916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.577895    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.578163 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.578143    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 15 16:27:26.578235 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.578206    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 15 16:27:26.578329 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.578314    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-xqppn\""
Sep 15 16:27:26.578387 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.578321    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.591022 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.590992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-bound-sa-token\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.591224 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.591050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8df29ac7-55fb-42c0-bec2-f5375a8543ac-ca-trust-extracted\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.591224 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.591106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-image-registry-private-configuration\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.591224 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.591162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrg28\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-kube-api-access-hrg28\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.591224 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.591209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44j5f\" (UniqueName: \"kubernetes.io/projected/f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6-kube-api-access-44j5f\") pod \"volume-data-source-validator-86695b7cc7-d7mdp\" (UID: \"f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp"
Sep 15 16:27:26.591431 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.591298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.591431 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.591349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-certificates\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.591431 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.591373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-installation-pull-secrets\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.591431 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.591397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-trusted-ca\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.595085 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.595057    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7f5d8767b4-qhwn7"]
Sep 15 16:27:26.595210 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.595060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.601304 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.601281    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.601498 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.601288    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.601605 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.601369    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x7vpq\""
Sep 15 16:27:26.603769 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.603748    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 15 16:27:26.605921 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.605905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 15 16:27:26.607188 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.607172    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"]
Sep 15 16:27:26.607318 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.607302    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.616116 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.616074    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 15 16:27:26.616116 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.616079    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.616116 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.616101    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 15 16:27:26.616371 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.616135    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.616371 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.616358    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 15 16:27:26.616448 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.616406    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 15 16:27:26.617316 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.617298    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-njfkt\""
Sep 15 16:27:26.623241 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.623222    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-s5lrt"]
Sep 15 16:27:26.623394 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.623375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.628444 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.628415    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 15 16:27:26.628444 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.628443    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.628616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.628459    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 15 16:27:26.628616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.628487    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.632805 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.632783    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-7txl7\""
Sep 15 16:27:26.640320 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.640299    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp"]
Sep 15 16:27:26.640479 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.640466    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9b4d"]
Sep 15 16:27:26.640620 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.640600    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"]
Sep 15 16:27:26.640726 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.640632    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-nhpth"]
Sep 15 16:27:26.640726 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.640651    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"]
Sep 15 16:27:26.640726 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.640676    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"]
Sep 15 16:27:26.640726 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.640699    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-25zwg"]
Sep 15 16:27:26.641019 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.640998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.645637 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.644922    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 15 16:27:26.645637 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.645228    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 15 16:27:26.646526 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.646034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.649709 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.647187    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.654135 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.654102    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-p7m4r\""
Sep 15 16:27:26.655630 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.655611    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 15 16:27:26.659738 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.659720    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-drgp5"]
Sep 15 16:27:26.659887 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.659867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-25zwg"
Sep 15 16:27:26.662464 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.662446    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-gph6n\""
Sep 15 16:27:26.662617 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.662598    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.662975 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.662957    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.675190 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.675170    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"]
Sep 15 16:27:26.675294 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.675208    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-s5lrt"]
Sep 15 16:27:26.675294 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.675231    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f5d8767b4-qhwn7"]
Sep 15 16:27:26.675294 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.675249    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-25zwg"]
Sep 15 16:27:26.675294 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.675273    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drgp5"]
Sep 15 16:27:26.675294 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.675274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drgp5"
Sep 15 16:27:26.677984 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.677925    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nn9ff\""
Sep 15 16:27:26.678091 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.677989    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 15 16:27:26.678091 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.677997    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 15 16:27:26.679526 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.679510    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 15 16:27:26.692213 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcn65\" (UniqueName: \"kubernetes.io/projected/552d2031-4b9e-477f-aff2-09b7ba12f053-kube-api-access-dcn65\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:26.692330 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692246    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.692330 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.692429 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-certificates\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.692478 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-trusted-ca\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.692585 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-tmp\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.692662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-snapshots\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.692698 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-trusted-ca-bundle\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.692730 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.692762 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.692795 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc88p\" (UniqueName: \"kubernetes.io/projected/63e81776-6e13-4425-b7e1-a5f42ea0ce43-kube-api-access-jc88p\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.692824 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.692858 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-image-registry-private-configuration\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.692891 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-stats-auth\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.692922 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692903    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-serving-cert\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.692954 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-certificates\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.692989 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63e81776-6e13-4425-b7e1-a5f42ea0ce43-config-volume\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.692989 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.692981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:26.693046 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.693079 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqx5\" (UniqueName: \"kubernetes.io/projected/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-kube-api-access-gpqx5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.693130 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.693178 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-default-certificate\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.693178 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0e1009b7-c437-4e0f-801e-96e3106c9524-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.693270 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-installation-pull-secrets\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.693270 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6fgw\" (UniqueName: \"kubernetes.io/projected/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-kube-api-access-m6fgw\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.693270 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693239    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-bound-sa-token\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.693270 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.693256    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 16:27:26.693443 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.693277    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d87d77784-67zfs: secret "image-registry-tls" not found
Sep 15 16:27:26.693443 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.693332    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls podName:8df29ac7-55fb-42c0-bec2-f5375a8543ac nodeName:}" failed. No retries permitted until 2026-09-15 16:27:27.193314849 +0000 UTC m=+35.561140040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls") pod "image-registry-d87d77784-67zfs" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac") : secret "image-registry-tls" not found
Sep 15 16:27:26.693443 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693261    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-service-ca-bundle\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.693616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbbwb\" (UniqueName: \"kubernetes.io/projected/0e1009b7-c437-4e0f-801e-96e3106c9524-kube-api-access-mbbwb\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.693616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8df29ac7-55fb-42c0-bec2-f5375a8543ac-ca-trust-extracted\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.693616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj8gj\" (UniqueName: \"kubernetes.io/projected/6385b954-9257-40b7-a82b-d32c090cd40e-kube-api-access-wj8gj\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.693771 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrg28\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-kube-api-access-hrg28\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.693771 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44j5f\" (UniqueName: \"kubernetes.io/projected/f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6-kube-api-access-44j5f\") pod \"volume-data-source-validator-86695b7cc7-d7mdp\" (UID: \"f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp"
Sep 15 16:27:26.693771 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e81776-6e13-4425-b7e1-a5f42ea0ce43-tmp-dir\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.693908 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-trusted-ca\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.693908 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.693879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8df29ac7-55fb-42c0-bec2-f5375a8543ac-ca-trust-extracted\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.697483 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.697461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-installation-pull-secrets\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.697608 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.697466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-image-registry-private-configuration\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.708864 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.708826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44j5f\" (UniqueName: \"kubernetes.io/projected/f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6-kube-api-access-44j5f\") pod \"volume-data-source-validator-86695b7cc7-d7mdp\" (UID: \"f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp"
Sep 15 16:27:26.709951 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.709929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-bound-sa-token\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.710255 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.710217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrg28\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-kube-api-access-hrg28\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:26.794605 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-service-ca-bundle\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.794605 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbbwb\" (UniqueName: \"kubernetes.io/projected/0e1009b7-c437-4e0f-801e-96e3106c9524-kube-api-access-mbbwb\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.794903 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj8gj\" (UniqueName: \"kubernetes.io/projected/6385b954-9257-40b7-a82b-d32c090cd40e-kube-api-access-wj8gj\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.794903 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4bb97b-0349-4136-bb5e-2f4c089d367a-serving-cert\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.794903 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e81776-6e13-4425-b7e1-a5f42ea0ce43-tmp-dir\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.794903 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794843    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh2zk\" (UniqueName: \"kubernetes.io/projected/8f4bb97b-0349-4136-bb5e-2f4c089d367a-kube-api-access-fh2zk\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.794903 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcn65\" (UniqueName: \"kubernetes.io/projected/552d2031-4b9e-477f-aff2-09b7ba12f053-kube-api-access-dcn65\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:26.795162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/992ed8e3-ebc1-46b8-890f-9150130cf77d-cert\") pod \"ingress-canary-drgp5\" (UID: \"992ed8e3-ebc1-46b8-890f-9150130cf77d\") " pod="openshift-ingress-canary/ingress-canary-drgp5"
Sep 15 16:27:26.795162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.794975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.795162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.795162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/63e81776-6e13-4425-b7e1-a5f42ea0ce43-tmp-dir\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.795162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-tmp\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.795175    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-snapshots\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-trusted-ca-bundle\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.795241    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls podName:0e1009b7-c437-4e0f-801e-96e3106c9524 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:27.295222816 +0000 UTC m=+35.663048012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrzv9" (UID: "0e1009b7-c437-4e0f-801e-96e3106c9524") : secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75fc8f59-26b4-4890-aa40-0ca7c847967f-serving-cert\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795273    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-service-ca-bundle\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc88p\" (UniqueName: \"kubernetes.io/projected/63e81776-6e13-4425-b7e1-a5f42ea0ce43-kube-api-access-jc88p\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.795400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795403    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-stats-auth\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-tmp\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-serving-cert\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f4bb97b-0349-4136-bb5e-2f4c089d367a-trusted-ca\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qrjb\" (UniqueName: \"kubernetes.io/projected/79abacbb-141b-4574-91df-121049c7477e-kube-api-access-7qrjb\") pod \"network-check-source-6f8b768455-25zwg\" (UID: \"79abacbb-141b-4574-91df-121049c7477e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-25zwg"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63e81776-6e13-4425-b7e1-a5f42ea0ce43-config-volume\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqx5\" (UniqueName: \"kubernetes.io/projected/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-kube-api-access-gpqx5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-default-certificate\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0e1009b7-c437-4e0f-801e-96e3106c9524-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795836    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-snapshots\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.795916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4bb97b-0349-4136-bb5e-2f4c089d367a-config\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.795956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m86k\" (UniqueName: \"kubernetes.io/projected/992ed8e3-ebc1-46b8-890f-9150130cf77d-kube-api-access-7m86k\") pod \"ingress-canary-drgp5\" (UID: \"992ed8e3-ebc1-46b8-890f-9150130cf77d\") " pod="openshift-ingress-canary/ingress-canary-drgp5"
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.795451    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.796005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-trusted-ca-bundle\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.795499    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.795737    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.795957    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:27.29594288 +0000 UTC m=+35.663768060 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : configmap references non-existent config key: service-ca.crt
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.796135    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:27.296118444 +0000 UTC m=+35.663943621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : secret "router-metrics-certs-default" not found
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.796172    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls podName:63e81776-6e13-4425-b7e1-a5f42ea0ce43 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:27.296161371 +0000 UTC m=+35.663986553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls") pod "dns-default-l9b4d" (UID: "63e81776-6e13-4425-b7e1-a5f42ea0ce43") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:26.796186    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls podName:552d2031-4b9e-477f-aff2-09b7ba12f053 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:27.296178496 +0000 UTC m=+35.664003676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8gs62" (UID: "552d2031-4b9e-477f-aff2-09b7ba12f053") : secret "samples-operator-tls" not found
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.796193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/63e81776-6e13-4425-b7e1-a5f42ea0ce43-config-volume\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.796202    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6fgw\" (UniqueName: \"kubernetes.io/projected/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-kube-api-access-m6fgw\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.796232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75fc8f59-26b4-4890-aa40-0ca7c847967f-config\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.796551 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.796257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57hcz\" (UniqueName: \"kubernetes.io/projected/75fc8f59-26b4-4890-aa40-0ca7c847967f-kube-api-access-57hcz\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.797239 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.796628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0e1009b7-c437-4e0f-801e-96e3106c9524-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.798273 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.798247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.798617 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.798596    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-stats-auth\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.799018 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.798992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-default-certificate\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.807767 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.807740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbbwb\" (UniqueName: \"kubernetes.io/projected/0e1009b7-c437-4e0f-801e-96e3106c9524-kube-api-access-mbbwb\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:26.808093 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.808074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj8gj\" (UniqueName: \"kubernetes.io/projected/6385b954-9257-40b7-a82b-d32c090cd40e-kube-api-access-wj8gj\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:26.808435 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.808415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-serving-cert\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.811481 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.811423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6fgw\" (UniqueName: \"kubernetes.io/projected/8f85e488-5160-4a79-a4b9-cf5c3c7734b5-kube-api-access-m6fgw\") pod \"insights-operator-5844b856db-nhpth\" (UID: \"8f85e488-5160-4a79-a4b9-cf5c3c7734b5\") " pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.811946 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.811920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc88p\" (UniqueName: \"kubernetes.io/projected/63e81776-6e13-4425-b7e1-a5f42ea0ce43-kube-api-access-jc88p\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:26.813476 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.813451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcn65\" (UniqueName: \"kubernetes.io/projected/552d2031-4b9e-477f-aff2-09b7ba12f053-kube-api-access-dcn65\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:26.813646 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.813622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqx5\" (UniqueName: \"kubernetes.io/projected/537e42cc-e1d6-4cb1-80d4-fd0209023ec9-kube-api-access-gpqx5\") pod \"kube-storage-version-migrator-operator-6c8f84f897-n8wht\" (UID: \"537e42cc-e1d6-4cb1-80d4-fd0209023ec9\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.819913 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.819892    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp"
Sep 15 16:27:26.865366 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.865329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-nhpth"
Sep 15 16:27:26.884786 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.884754    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"
Sep 15 16:27:26.897101 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4bb97b-0349-4136-bb5e-2f4c089d367a-config\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.897221 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m86k\" (UniqueName: \"kubernetes.io/projected/992ed8e3-ebc1-46b8-890f-9150130cf77d-kube-api-access-7m86k\") pod \"ingress-canary-drgp5\" (UID: \"992ed8e3-ebc1-46b8-890f-9150130cf77d\") " pod="openshift-ingress-canary/ingress-canary-drgp5"
Sep 15 16:27:26.897221 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75fc8f59-26b4-4890-aa40-0ca7c847967f-config\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.897221 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57hcz\" (UniqueName: \"kubernetes.io/projected/75fc8f59-26b4-4890-aa40-0ca7c847967f-kube-api-access-57hcz\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.897376 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4bb97b-0349-4136-bb5e-2f4c089d367a-serving-cert\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.897376 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh2zk\" (UniqueName: \"kubernetes.io/projected/8f4bb97b-0349-4136-bb5e-2f4c089d367a-kube-api-access-fh2zk\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.897479 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/992ed8e3-ebc1-46b8-890f-9150130cf77d-cert\") pod \"ingress-canary-drgp5\" (UID: \"992ed8e3-ebc1-46b8-890f-9150130cf77d\") " pod="openshift-ingress-canary/ingress-canary-drgp5"
Sep 15 16:27:26.897479 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75fc8f59-26b4-4890-aa40-0ca7c847967f-serving-cert\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.897599 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f4bb97b-0349-4136-bb5e-2f4c089d367a-trusted-ca\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.897599 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qrjb\" (UniqueName: \"kubernetes.io/projected/79abacbb-141b-4574-91df-121049c7477e-kube-api-access-7qrjb\") pod \"network-check-source-6f8b768455-25zwg\" (UID: \"79abacbb-141b-4574-91df-121049c7477e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-25zwg"
Sep 15 16:27:26.897775 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/75fc8f59-26b4-4890-aa40-0ca7c847967f-config\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.897921 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.897871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8f4bb97b-0349-4136-bb5e-2f4c089d367a-config\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.898440 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.898420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8f4bb97b-0349-4136-bb5e-2f4c089d367a-trusted-ca\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.900026 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.899997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/75fc8f59-26b4-4890-aa40-0ca7c847967f-serving-cert\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.900287 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.900270    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f4bb97b-0349-4136-bb5e-2f4c089d367a-serving-cert\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.900404 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.900384    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/992ed8e3-ebc1-46b8-890f-9150130cf77d-cert\") pod \"ingress-canary-drgp5\" (UID: \"992ed8e3-ebc1-46b8-890f-9150130cf77d\") " pod="openshift-ingress-canary/ingress-canary-drgp5"
Sep 15 16:27:26.907698 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.907667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh2zk\" (UniqueName: \"kubernetes.io/projected/8f4bb97b-0349-4136-bb5e-2f4c089d367a-kube-api-access-fh2zk\") pod \"console-operator-696884fb9c-s5lrt\" (UID: \"8f4bb97b-0349-4136-bb5e-2f4c089d367a\") " pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.907860 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.907822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57hcz\" (UniqueName: \"kubernetes.io/projected/75fc8f59-26b4-4890-aa40-0ca7c847967f-kube-api-access-57hcz\") pod \"service-ca-operator-cf7c776b7-zj6qn\" (UID: \"75fc8f59-26b4-4890-aa40-0ca7c847967f\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.908169 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.908153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m86k\" (UniqueName: \"kubernetes.io/projected/992ed8e3-ebc1-46b8-890f-9150130cf77d-kube-api-access-7m86k\") pod \"ingress-canary-drgp5\" (UID: \"992ed8e3-ebc1-46b8-890f-9150130cf77d\") " pod="openshift-ingress-canary/ingress-canary-drgp5"
Sep 15 16:27:26.908739 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.908721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qrjb\" (UniqueName: \"kubernetes.io/projected/79abacbb-141b-4574-91df-121049c7477e-kube-api-access-7qrjb\") pod \"network-check-source-6f8b768455-25zwg\" (UID: \"79abacbb-141b-4574-91df-121049c7477e\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-25zwg"
Sep 15 16:27:26.932787 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.932749    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"
Sep 15 16:27:26.956951 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.956920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:26.968636 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.968607    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-25zwg"
Sep 15 16:27:26.986002 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:26.985976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-drgp5"
Sep 15 16:27:27.153607 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.153517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:27.153607 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.153523    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:27.153807 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.153643    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:27.157826 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.157803    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 16:27:27.157989 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.157969    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhmxd\""
Sep 15 16:27:27.158846 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.158824    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 16:27:27.158935 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.158864    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkdt8\""
Sep 15 16:27:27.200893 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.200869    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:27.201406 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.201379    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 16:27:27.201406 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.201407    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d87d77784-67zfs: secret "image-registry-tls" not found
Sep 15 16:27:27.201594 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.201476    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls podName:8df29ac7-55fb-42c0-bec2-f5375a8543ac nodeName:}" failed. No retries permitted until 2026-09-15 16:27:28.201457037 +0000 UTC m=+36.569282231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls") pod "image-registry-d87d77784-67zfs" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac") : secret "image-registry-tls" not found
Sep 15 16:27:27.302438 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.302397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:27.302593 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.302481    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:27.302593 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.302558    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 16:27:27.302701 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.302595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:27.302701 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.302663    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:28.302640024 +0000 UTC m=+36.670465226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : secret "router-metrics-certs-default" not found
Sep 15 16:27:27.302701 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.302689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:27.303002 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.302983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:27.303371 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.303160    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:27.303371 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.303179    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 16:27:27.303371 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.303237    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls podName:552d2031-4b9e-477f-aff2-09b7ba12f053 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:28.303220977 +0000 UTC m=+36.671046178 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8gs62" (UID: "552d2031-4b9e-477f-aff2-09b7ba12f053") : secret "samples-operator-tls" not found
Sep 15 16:27:27.303371 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.303258    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls podName:0e1009b7-c437-4e0f-801e-96e3106c9524 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:28.303247715 +0000 UTC m=+36.671072903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrzv9" (UID: "0e1009b7-c437-4e0f-801e-96e3106c9524") : secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:27.303371 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.303317    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:27.303371 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.303322    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:28.303275959 +0000 UTC m=+36.671101155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : configmap references non-existent config key: service-ca.crt
Sep 15 16:27:27.303371 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:27.303352    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls podName:63e81776-6e13-4425-b7e1-a5f42ea0ce43 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:28.303341435 +0000 UTC m=+36.671166628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls") pod "dns-default-l9b4d" (UID: "63e81776-6e13-4425-b7e1-a5f42ea0ce43") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:27.627877 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.627636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-25zwg"]
Sep 15 16:27:27.630351 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:27.630325    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79abacbb_141b_4574_91df_121049c7477e.slice/crio-88d656811fe615a4ccced08b107c8eb77702436d8d0dcdaa315fb65ae7cf0a29 WatchSource:0}: Error finding container 88d656811fe615a4ccced08b107c8eb77702436d8d0dcdaa315fb65ae7cf0a29: Status 404 returned error can't find the container with id 88d656811fe615a4ccced08b107c8eb77702436d8d0dcdaa315fb65ae7cf0a29
Sep 15 16:27:27.637187 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.637168    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn"]
Sep 15 16:27:27.639487 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:27.639462    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75fc8f59_26b4_4890_aa40_0ca7c847967f.slice/crio-5203afdae61997d8c12ca6d3166ff33d58d49b8ac2df7e64b18d0631868c9fd4 WatchSource:0}: Error finding container 5203afdae61997d8c12ca6d3166ff33d58d49b8ac2df7e64b18d0631868c9fd4: Status 404 returned error can't find the container with id 5203afdae61997d8c12ca6d3166ff33d58d49b8ac2df7e64b18d0631868c9fd4
Sep 15 16:27:27.644785 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.644710    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp"]
Sep 15 16:27:27.646991 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.645749    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-drgp5"]
Sep 15 16:27:27.647816 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.647611    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-nhpth"]
Sep 15 16:27:27.648774 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.648452    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht"]
Sep 15 16:27:27.649892 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:27.649848    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90f3bef_0bf8_4c28_8cd1_a464eddbc0f6.slice/crio-b0812242292956fa0cafcbf8066c4df8a673d7032145b6310f7126c2a252fb5b WatchSource:0}: Error finding container b0812242292956fa0cafcbf8066c4df8a673d7032145b6310f7126c2a252fb5b: Status 404 returned error can't find the container with id b0812242292956fa0cafcbf8066c4df8a673d7032145b6310f7126c2a252fb5b
Sep 15 16:27:27.649892 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:27.649855    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-s5lrt"]
Sep 15 16:27:27.651368 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:27.651215    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f85e488_5160_4a79_a4b9_cf5c3c7734b5.slice/crio-f0af1b408f9a3449b9b5eaf7fc30ade2c1bbdadc6eaa180e8df47e454144308f WatchSource:0}: Error finding container f0af1b408f9a3449b9b5eaf7fc30ade2c1bbdadc6eaa180e8df47e454144308f: Status 404 returned error can't find the container with id f0af1b408f9a3449b9b5eaf7fc30ade2c1bbdadc6eaa180e8df47e454144308f
Sep 15 16:27:27.663001 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:27.662853    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod537e42cc_e1d6_4cb1_80d4_fd0209023ec9.slice/crio-7399de3b2a0f9f63ca102d96f4e8fe5179ac7f618ac9ee8632e279497843e0f5 WatchSource:0}: Error finding container 7399de3b2a0f9f63ca102d96f4e8fe5179ac7f618ac9ee8632e279497843e0f5: Status 404 returned error can't find the container with id 7399de3b2a0f9f63ca102d96f4e8fe5179ac7f618ac9ee8632e279497843e0f5
Sep 15 16:27:27.663124 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:27.663093    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992ed8e3_ebc1_46b8_890f_9150130cf77d.slice/crio-3d5d287705651c86549dbaadffce5ed2b2dfb0ff1dfd396ca7f0b4a6a1fa681e WatchSource:0}: Error finding container 3d5d287705651c86549dbaadffce5ed2b2dfb0ff1dfd396ca7f0b4a6a1fa681e: Status 404 returned error can't find the container with id 3d5d287705651c86549dbaadffce5ed2b2dfb0ff1dfd396ca7f0b4a6a1fa681e
Sep 15 16:27:27.664100 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:27.664073    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4bb97b_0349_4136_bb5e_2f4c089d367a.slice/crio-02c7a24d13afecfc131a936167a937b46c43e0ca92726029dc3af15e08e3c673 WatchSource:0}: Error finding container 02c7a24d13afecfc131a936167a937b46c43e0ca92726029dc3af15e08e3c673: Status 404 returned error can't find the container with id 02c7a24d13afecfc131a936167a937b46c43e0ca92726029dc3af15e08e3c673
Sep 15 16:27:28.211540 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.211502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:28.212272 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.211847    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 16:27:28.212272 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.211871    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d87d77784-67zfs: secret "image-registry-tls" not found
Sep 15 16:27:28.212272 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.211937    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls podName:8df29ac7-55fb-42c0-bec2-f5375a8543ac nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.211917917 +0000 UTC m=+38.579743114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls") pod "image-registry-d87d77784-67zfs" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac") : secret "image-registry-tls" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.312478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.312526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.312589    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.312621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.312749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.312878    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.312943    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls podName:0e1009b7-c437-4e0f-801e-96e3106c9524 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.312925358 +0000 UTC m=+38.680750549 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrzv9" (UID: "0e1009b7-c437-4e0f-801e-96e3106c9524") : secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.313337    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.313386    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.313372363 +0000 UTC m=+38.681197554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : secret "router-metrics-certs-default" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.313451    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.313481    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls podName:63e81776-6e13-4425-b7e1-a5f42ea0ce43 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.313469709 +0000 UTC m=+38.681294903 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls") pod "dns-default-l9b4d" (UID: "63e81776-6e13-4425-b7e1-a5f42ea0ce43") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.313536    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.313589    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls podName:552d2031-4b9e-477f-aff2-09b7ba12f053 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.313557049 +0000 UTC m=+38.681382241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8gs62" (UID: "552d2031-4b9e-477f-aff2-09b7ba12f053") : secret "samples-operator-tls" not found
Sep 15 16:27:28.313697 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:28.313659    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:30.313648856 +0000 UTC m=+38.681474057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : configmap references non-existent config key: service-ca.crt
Sep 15 16:27:28.358240 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.358202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-nhpth" event={"ID":"8f85e488-5160-4a79-a4b9-cf5c3c7734b5","Type":"ContainerStarted","Data":"f0af1b408f9a3449b9b5eaf7fc30ade2c1bbdadc6eaa180e8df47e454144308f"}
Sep 15 16:27:28.361769 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.361737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-25zwg" event={"ID":"79abacbb-141b-4574-91df-121049c7477e","Type":"ContainerStarted","Data":"88d656811fe615a4ccced08b107c8eb77702436d8d0dcdaa315fb65ae7cf0a29"}
Sep 15 16:27:28.363698 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.363624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn" event={"ID":"75fc8f59-26b4-4890-aa40-0ca7c847967f","Type":"ContainerStarted","Data":"5203afdae61997d8c12ca6d3166ff33d58d49b8ac2df7e64b18d0631868c9fd4"}
Sep 15 16:27:28.365922 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.365863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" event={"ID":"8f4bb97b-0349-4136-bb5e-2f4c089d367a","Type":"ContainerStarted","Data":"02c7a24d13afecfc131a936167a937b46c43e0ca92726029dc3af15e08e3c673"}
Sep 15 16:27:28.367800 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.367731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp" event={"ID":"f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6","Type":"ContainerStarted","Data":"b0812242292956fa0cafcbf8066c4df8a673d7032145b6310f7126c2a252fb5b"}
Sep 15 16:27:28.377814 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.377660    2566 generic.go:356] "Generic (PLEG): container finished" podID="35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1" containerID="6becb254805145de0ba12698eced7b1cf0f7c22223118a69ad1436e18f4ad5ca" exitCode=0
Sep 15 16:27:28.377814 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.377760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerDied","Data":"6becb254805145de0ba12698eced7b1cf0f7c22223118a69ad1436e18f4ad5ca"}
Sep 15 16:27:28.381025 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.380916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht" event={"ID":"537e42cc-e1d6-4cb1-80d4-fd0209023ec9","Type":"ContainerStarted","Data":"7399de3b2a0f9f63ca102d96f4e8fe5179ac7f618ac9ee8632e279497843e0f5"}
Sep 15 16:27:28.391781 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:28.391701    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drgp5" event={"ID":"992ed8e3-ebc1-46b8-890f-9150130cf77d","Type":"ContainerStarted","Data":"3d5d287705651c86549dbaadffce5ed2b2dfb0ff1dfd396ca7f0b4a6a1fa681e"}
Sep 15 16:27:29.407445 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:29.406475    2566 generic.go:356] "Generic (PLEG): container finished" podID="35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1" containerID="c9f71ed1a12faac956baa4b950882ba45d9fe4db97edc88175da0b4c849f0484" exitCode=0
Sep 15 16:27:29.407445 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:29.406563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerDied","Data":"c9f71ed1a12faac956baa4b950882ba45d9fe4db97edc88175da0b4c849f0484"}
Sep 15 16:27:30.234199 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:30.234098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:30.234398 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.234240    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 16:27:30.234398 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.234257    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d87d77784-67zfs: secret "image-registry-tls" not found
Sep 15 16:27:30.234398 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.234318    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls podName:8df29ac7-55fb-42c0-bec2-f5375a8543ac nodeName:}" failed. No retries permitted until 2026-09-15 16:27:34.234299937 +0000 UTC m=+42.602125128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls") pod "image-registry-d87d77784-67zfs" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac") : secret "image-registry-tls" not found
Sep 15 16:27:30.335310 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:30.335248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:30.335310 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:30.335313    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:30.335538 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:30.335340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:30.335538 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:30.335384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:30.335538 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:30.335412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:30.335710 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.335649    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:34.335626337 +0000 UTC m=+42.703451531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : configmap references non-existent config key: service-ca.crt
Sep 15 16:27:30.336218 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.335930    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:30.336218 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.335997    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls podName:63e81776-6e13-4425-b7e1-a5f42ea0ce43 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:34.335980248 +0000 UTC m=+42.703805443 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls") pod "dns-default-l9b4d" (UID: "63e81776-6e13-4425-b7e1-a5f42ea0ce43") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:30.336218 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.336025    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 16:27:30.336218 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.336051    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:30.336218 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.336061    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls podName:552d2031-4b9e-477f-aff2-09b7ba12f053 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:34.336049348 +0000 UTC m=+42.703874547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8gs62" (UID: "552d2031-4b9e-477f-aff2-09b7ba12f053") : secret "samples-operator-tls" not found
Sep 15 16:27:30.336218 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.336084    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls podName:0e1009b7-c437-4e0f-801e-96e3106c9524 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:34.336073711 +0000 UTC m=+42.703898901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrzv9" (UID: "0e1009b7-c437-4e0f-801e-96e3106c9524") : secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:30.336218 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.336119    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 16:27:30.336218 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:30.336149    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:34.336139799 +0000 UTC m=+42.703964991 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : secret "router-metrics-certs-default" not found
Sep 15 16:27:34.270777 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:34.270693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:34.271286 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.270847    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 16:27:34.271286 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.270864    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d87d77784-67zfs: secret "image-registry-tls" not found
Sep 15 16:27:34.271286 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.270929    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls podName:8df29ac7-55fb-42c0-bec2-f5375a8543ac nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.270907235 +0000 UTC m=+50.638732436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls") pod "image-registry-d87d77784-67zfs" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac") : secret "image-registry-tls" not found
Sep 15 16:27:34.371956 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:34.371921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:34.372142 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:34.371987    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:34.372142 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:34.372011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:34.372142 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:34.372038    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:34.372142 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:34.372066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:34.372142 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372071    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:34.372142 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372141    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 16:27:34.372459 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372148    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 16:27:34.372459 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372144    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls podName:0e1009b7-c437-4e0f-801e-96e3106c9524 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.37212544 +0000 UTC m=+50.739950634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrzv9" (UID: "0e1009b7-c437-4e0f-801e-96e3106c9524") : secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:34.372459 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372195    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.372180217 +0000 UTC m=+50.740005412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : secret "router-metrics-certs-default" not found
Sep 15 16:27:34.372459 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372220    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:34.372459 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372260    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls podName:63e81776-6e13-4425-b7e1-a5f42ea0ce43 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.372247354 +0000 UTC m=+50.740072549 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls") pod "dns-default-l9b4d" (UID: "63e81776-6e13-4425-b7e1-a5f42ea0ce43") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:34.372459 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372278    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls podName:552d2031-4b9e-477f-aff2-09b7ba12f053 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.372268647 +0000 UTC m=+50.740093831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8gs62" (UID: "552d2031-4b9e-477f-aff2-09b7ba12f053") : secret "samples-operator-tls" not found
Sep 15 16:27:34.372459 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:34.372291    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:42.372283959 +0000 UTC m=+50.740109137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : configmap references non-existent config key: service-ca.crt
Sep 15 16:27:36.427278 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.427242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn" event={"ID":"75fc8f59-26b4-4890-aa40-0ca7c847967f","Type":"ContainerStarted","Data":"b5d1c1ae592f0381d9c5b90504670aa72511918b870e0be57a79bd4c402fd5be"}
Sep 15 16:27:36.429053 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.429029    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/0.log"
Sep 15 16:27:36.429154 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.429074    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f4bb97b-0349-4136-bb5e-2f4c089d367a" containerID="1bed0a10ccb1b0f44d28af702d592fd57fab07b56ce82dece2b542fde7d41dce" exitCode=255
Sep 15 16:27:36.429154 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.429145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" event={"ID":"8f4bb97b-0349-4136-bb5e-2f4c089d367a","Type":"ContainerDied","Data":"1bed0a10ccb1b0f44d28af702d592fd57fab07b56ce82dece2b542fde7d41dce"}
Sep 15 16:27:36.429352 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.429336    2566 scope.go:117] "RemoveContainer" containerID="1bed0a10ccb1b0f44d28af702d592fd57fab07b56ce82dece2b542fde7d41dce"
Sep 15 16:27:36.431446 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.431058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp" event={"ID":"f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6","Type":"ContainerStarted","Data":"c0740fdba1161cd09f8f8aa7a4ce7f9d082689e718cd500d46d41d59578a0031"}
Sep 15 16:27:36.435466 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.435446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g4trk" event={"ID":"35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1","Type":"ContainerStarted","Data":"debcc8beb92119c89833651d69058384b54aa22c9b32783e21091b40f9acf95c"}
Sep 15 16:27:36.437105 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.437078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht" event={"ID":"537e42cc-e1d6-4cb1-80d4-fd0209023ec9","Type":"ContainerStarted","Data":"904875b9406bcd0b849692b56993a9f857524e286ac313346b54749bb3b7cee6"}
Sep 15 16:27:36.438697 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.438676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-drgp5" event={"ID":"992ed8e3-ebc1-46b8-890f-9150130cf77d","Type":"ContainerStarted","Data":"a8b4842656b09e925c342c89410deea7682f0fb09a4d255a53e47689c644b664"}
Sep 15 16:27:36.440240 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.440198    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-nhpth" event={"ID":"8f85e488-5160-4a79-a4b9-cf5c3c7734b5","Type":"ContainerStarted","Data":"3712400b416411186f234d7c02d0b2c4283c1410ad1fb55d2d5e7c05aa802a7e"}
Sep 15 16:27:36.441753 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.441732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-25zwg" event={"ID":"79abacbb-141b-4574-91df-121049c7477e","Type":"ContainerStarted","Data":"e0c26215db0935be0b7df7b7107f654e36baa565c47eea0e9e7c668949fbcb7f"}
Sep 15 16:27:36.447980 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.447939    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn" podStartSLOduration=4.655119696 podStartE2EDuration="12.447926035s" totalImagesPullingTime="7.792806339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:36.446710017 +0000 UTC m=+44.814535218" watchObservedRunningTime="2026-09-15 16:27:36.447926035 +0000 UTC m=+44.815751233"
Sep 15 16:27:36.477794 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.477689    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g4trk" podStartSLOduration=13.451587732 podStartE2EDuration="43.477669659s" totalImagesPullingTime="30.026081927s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:36.476156954 +0000 UTC m=+44.843982154" watchObservedRunningTime="2026-09-15 16:27:36.477669659 +0000 UTC m=+44.845494859"
Sep 15 16:27:36.501053 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.500991    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht" podStartSLOduration=4.730583176 podStartE2EDuration="12.500970434s" totalImagesPullingTime="7.770387258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:36.499907011 +0000 UTC m=+44.867732211" watchObservedRunningTime="2026-09-15 16:27:36.500970434 +0000 UTC m=+44.868795635"
Sep 15 16:27:36.538270 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.537011    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-drgp5" podStartSLOduration=2.769208285 podStartE2EDuration="10.536994568s" totalImagesPullingTime="7.767786283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:36.535340921 +0000 UTC m=+44.903166120" watchObservedRunningTime="2026-09-15 16:27:36.536994568 +0000 UTC m=+44.904819767"
Sep 15 16:27:36.592352 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.592295    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-d7mdp" podStartSLOduration=5.864128484 podStartE2EDuration="13.592274996s" totalImagesPullingTime="7.728146512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:36.557258857 +0000 UTC m=+44.925084049" watchObservedRunningTime="2026-09-15 16:27:36.592274996 +0000 UTC m=+44.960100196"
Sep 15 16:27:36.630900 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.630844    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-nhpth" podStartSLOduration=5.847283889 podStartE2EDuration="13.630824046s" totalImagesPullingTime="7.783540157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:36.592266676 +0000 UTC m=+44.960091898" watchObservedRunningTime="2026-09-15 16:27:36.630824046 +0000 UTC m=+44.998649247"
Sep 15 16:27:36.659801 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.659754    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-25zwg" podStartSLOduration=3.845448498 podStartE2EDuration="11.659737037s" totalImagesPullingTime="7.814288539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:36.659092633 +0000 UTC m=+45.026917833" watchObservedRunningTime="2026-09-15 16:27:36.659737037 +0000 UTC m=+45.027562237"
Sep 15 16:27:36.957277 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.957188    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:36.957436 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:36.957285    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:37.375602 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.375512    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ffhfm"]
Sep 15 16:27:37.399289 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.399260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.402267 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.402242    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 15 16:27:37.402714 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.402696    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wmprm\""
Sep 15 16:27:37.402813 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.402702    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 15 16:27:37.426330 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.426305    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ffhfm"]
Sep 15 16:27:37.446547 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.446528    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/1.log"
Sep 15 16:27:37.446954 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.446938    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/0.log"
Sep 15 16:27:37.447032 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.446978    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f4bb97b-0349-4136-bb5e-2f4c089d367a" containerID="67ac9c7c046a3ab2dc58f014d29387c9a133b8ea5db551fe40005f87779d994c" exitCode=255
Sep 15 16:27:37.447120 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.447087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" event={"ID":"8f4bb97b-0349-4136-bb5e-2f4c089d367a","Type":"ContainerDied","Data":"67ac9c7c046a3ab2dc58f014d29387c9a133b8ea5db551fe40005f87779d994c"}
Sep 15 16:27:37.447176 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.447155    2566 scope.go:117] "RemoveContainer" containerID="1bed0a10ccb1b0f44d28af702d592fd57fab07b56ce82dece2b542fde7d41dce"
Sep 15 16:27:37.447237 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.447224    2566 scope.go:117] "RemoveContainer" containerID="67ac9c7c046a3ab2dc58f014d29387c9a133b8ea5db551fe40005f87779d994c"
Sep 15 16:27:37.447448 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:37.447410    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-s5lrt_openshift-console-operator(8f4bb97b-0349-4136-bb5e-2f4c089d367a)\"" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" podUID="8f4bb97b-0349-4136-bb5e-2f4c089d367a"
Sep 15 16:27:37.504057 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.503987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.504057 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.504025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.504267 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.504132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znzqx\" (UniqueName: \"kubernetes.io/projected/256c5634-bc43-4851-9efb-81dc294aeb42-kube-api-access-znzqx\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.504267 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.504173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/256c5634-bc43-4851-9efb-81dc294aeb42-data-volume\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.504623 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.504602    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/256c5634-bc43-4851-9efb-81dc294aeb42-crio-socket\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.605330 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.605299    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/256c5634-bc43-4851-9efb-81dc294aeb42-crio-socket\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.605496 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.605482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.605548 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.605498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/256c5634-bc43-4851-9efb-81dc294aeb42-crio-socket\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.605548 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.605506    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.605640 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.605586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znzqx\" (UniqueName: \"kubernetes.io/projected/256c5634-bc43-4851-9efb-81dc294aeb42-kube-api-access-znzqx\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.605640 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:37.605607    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:37.605640 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.605619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/256c5634-bc43-4851-9efb-81dc294aeb42-data-volume\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.605741 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:37.605664    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:38.1056511 +0000 UTC m=+46.473476282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:37.605741 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:37.605608    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 16:27:37.605820 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:37.605741    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:38.105728059 +0000 UTC m=+46.473553251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : configmap "kube-rbac-proxy" not found
Sep 15 16:27:37.605893 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.605879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/256c5634-bc43-4851-9efb-81dc294aeb42-data-volume\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:37.623819 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:37.623793    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znzqx\" (UniqueName: \"kubernetes.io/projected/256c5634-bc43-4851-9efb-81dc294aeb42-kube-api-access-znzqx\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:38.109066 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:38.109029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:38.109066 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:38.109065    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:38.109337 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:38.109157    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 16:27:38.109337 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:38.109210    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:38.109337 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:38.109253    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:39.109236456 +0000 UTC m=+47.477061646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : configmap "kube-rbac-proxy" not found
Sep 15 16:27:38.109337 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:38.109268    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:39.109262111 +0000 UTC m=+47.477087288 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:38.455291 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:38.455264    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/1.log"
Sep 15 16:27:38.455666 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:38.455558    2566 scope.go:117] "RemoveContainer" containerID="67ac9c7c046a3ab2dc58f014d29387c9a133b8ea5db551fe40005f87779d994c"
Sep 15 16:27:38.455745 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:38.455726    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-s5lrt_openshift-console-operator(8f4bb97b-0349-4136-bb5e-2f4c089d367a)\"" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" podUID="8f4bb97b-0349-4136-bb5e-2f4c089d367a"
Sep 15 16:27:38.822641 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:38.822533    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bpcsp_55b80b8c-5e99-4d98-9193-0dd6056f00e5/dns-node-resolver/0.log"
Sep 15 16:27:39.118238 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:39.118157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:39.118238 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:39.118196    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:39.118451 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:39.118289    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 16:27:39.118451 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:39.118373    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:41.118349294 +0000 UTC m=+49.486174488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : configmap "kube-rbac-proxy" not found
Sep 15 16:27:39.118451 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:39.118395    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:39.118565 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:39.118490    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:41.118477219 +0000 UTC m=+49.486302412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:40.019791 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:40.019763    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvx5n_f64207aa-a010-4064-99ee-1ca2028b239e/node-ca/0.log"
Sep 15 16:27:40.622024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:40.621995    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drgp5_992ed8e3-ebc1-46b8-890f-9150130cf77d/serve-healthcheck-canary/0.log"
Sep 15 16:27:41.136380 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:41.136345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:41.136380 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:41.136381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:41.136792 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:41.136466    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:41.136792 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:41.136469    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 15 16:27:41.136792 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:41.136518    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:45.136506171 +0000 UTC m=+53.504331348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:41.136792 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:41.136538    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:45.136524317 +0000 UTC m=+53.504349498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : configmap "kube-rbac-proxy" not found
Sep 15 16:27:42.347334 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:42.347302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:42.347756 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.347456    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 15 16:27:42.347756 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.347474    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d87d77784-67zfs: secret "image-registry-tls" not found
Sep 15 16:27:42.347756 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.347527    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls podName:8df29ac7-55fb-42c0-bec2-f5375a8543ac nodeName:}" failed. No retries permitted until 2026-09-15 16:27:58.347511273 +0000 UTC m=+66.715336455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls") pod "image-registry-d87d77784-67zfs" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac") : secret "image-registry-tls" not found
Sep 15 16:27:42.447899 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:42.447854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:42.447899 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:42.447901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:42.448139 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448008    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:58.447994242 +0000 UTC m=+66.815819431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : configmap references non-existent config key: service-ca.crt
Sep 15 16:27:42.448139 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448019    2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 15 16:27:42.448139 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448085    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls podName:552d2031-4b9e-477f-aff2-09b7ba12f053 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:58.448067815 +0000 UTC m=+66.815893004 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-8gs62" (UID: "552d2031-4b9e-477f-aff2-09b7ba12f053") : secret "samples-operator-tls" not found
Sep 15 16:27:42.448308 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:42.448225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:42.448308 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:42.448277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:42.448404 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:42.448306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:42.448404 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448363    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:42.448475 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448403    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 15 16:27:42.448475 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448407    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 15 16:27:42.448475 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448412    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls podName:0e1009b7-c437-4e0f-801e-96e3106c9524 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:58.448398482 +0000 UTC m=+66.816223673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-lrzv9" (UID: "0e1009b7-c437-4e0f-801e-96e3106c9524") : secret "cluster-monitoring-operator-tls" not found
Sep 15 16:27:42.448475 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448467    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs podName:6385b954-9257-40b7-a82b-d32c090cd40e nodeName:}" failed. No retries permitted until 2026-09-15 16:27:58.448457971 +0000 UTC m=+66.816283149 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs") pod "router-default-7f5d8767b4-qhwn7" (UID: "6385b954-9257-40b7-a82b-d32c090cd40e") : secret "router-metrics-certs-default" not found
Sep 15 16:27:42.448634 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:42.448477    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls podName:63e81776-6e13-4425-b7e1-a5f42ea0ce43 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:58.448471917 +0000 UTC m=+66.816297093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls") pod "dns-default-l9b4d" (UID: "63e81776-6e13-4425-b7e1-a5f42ea0ce43") : secret "dns-default-metrics-tls" not found
Sep 15 16:27:45.175009 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:45.174970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:45.175009 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:45.175015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:45.175519 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:45.175116    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:45.175519 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:45.175176    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls podName:256c5634-bc43-4851-9efb-81dc294aeb42 nodeName:}" failed. No retries permitted until 2026-09-15 16:27:53.175162739 +0000 UTC m=+61.542987918 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ffhfm" (UID: "256c5634-bc43-4851-9efb-81dc294aeb42") : secret "insights-runtime-extractor-tls" not found
Sep 15 16:27:45.175621 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:45.175517    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/256c5634-bc43-4851-9efb-81dc294aeb42-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:46.957708 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:46.957669    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:46.958124 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:46.957736    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:46.958124 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:46.958052    2566 scope.go:117] "RemoveContainer" containerID="67ac9c7c046a3ab2dc58f014d29387c9a133b8ea5db551fe40005f87779d994c"
Sep 15 16:27:47.486423 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:47.486394    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:27:47.486878 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:47.486858    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/1.log"
Sep 15 16:27:47.486994 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:47.486897    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f4bb97b-0349-4136-bb5e-2f4c089d367a" containerID="b6b14dadd8c71c0d9548fa2262bf1196893452426c418fb8723e632c68830371" exitCode=255
Sep 15 16:27:47.486994 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:47.486966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" event={"ID":"8f4bb97b-0349-4136-bb5e-2f4c089d367a","Type":"ContainerDied","Data":"b6b14dadd8c71c0d9548fa2262bf1196893452426c418fb8723e632c68830371"}
Sep 15 16:27:47.487097 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:47.487020    2566 scope.go:117] "RemoveContainer" containerID="67ac9c7c046a3ab2dc58f014d29387c9a133b8ea5db551fe40005f87779d994c"
Sep 15 16:27:47.487371 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:47.487346    2566 scope.go:117] "RemoveContainer" containerID="b6b14dadd8c71c0d9548fa2262bf1196893452426c418fb8723e632c68830371"
Sep 15 16:27:47.487583 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:47.487530    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-s5lrt_openshift-console-operator(8f4bb97b-0349-4136-bb5e-2f4c089d367a)\"" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" podUID="8f4bb97b-0349-4136-bb5e-2f4c089d367a"
Sep 15 16:27:48.495130 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:48.495103    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:27:53.243471 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:53.243432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:53.245877 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:53.245852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/256c5634-bc43-4851-9efb-81dc294aeb42-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ffhfm\" (UID: \"256c5634-bc43-4851-9efb-81dc294aeb42\") " pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:53.311197 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:53.311170    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-wmprm\""
Sep 15 16:27:53.319184 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:53.319164    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ffhfm"
Sep 15 16:27:53.351640 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:53.351619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-sc5tl"
Sep 15 16:27:53.595507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:53.595434    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ffhfm"]
Sep 15 16:27:53.598752 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:53.598722    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod256c5634_bc43_4851_9efb_81dc294aeb42.slice/crio-ea0a5de445ce01be62acdd51b8f28a66c85482022fa62e08c0e3cd9bb04e8204 WatchSource:0}: Error finding container ea0a5de445ce01be62acdd51b8f28a66c85482022fa62e08c0e3cd9bb04e8204: Status 404 returned error can't find the container with id ea0a5de445ce01be62acdd51b8f28a66c85482022fa62e08c0e3cd9bb04e8204
Sep 15 16:27:54.513564 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:54.513480    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ffhfm" event={"ID":"256c5634-bc43-4851-9efb-81dc294aeb42","Type":"ContainerStarted","Data":"09c6c3244055329f726bcdbd510636108b821b9f52a72bec5e1bf89adcbd933b"}
Sep 15 16:27:54.513564 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:54.513538    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ffhfm" event={"ID":"256c5634-bc43-4851-9efb-81dc294aeb42","Type":"ContainerStarted","Data":"73301898f6cf01801487667ece579d83369ff9c169f9763ef342afb9f41c7640"}
Sep 15 16:27:54.513564 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:54.513558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ffhfm" event={"ID":"256c5634-bc43-4851-9efb-81dc294aeb42","Type":"ContainerStarted","Data":"ea0a5de445ce01be62acdd51b8f28a66c85482022fa62e08c0e3cd9bb04e8204"}
Sep 15 16:27:56.521895 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:56.521855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ffhfm" event={"ID":"256c5634-bc43-4851-9efb-81dc294aeb42","Type":"ContainerStarted","Data":"0f5a88e48c4f77adc06c0e38e546dcbb8706043c217e04045ae3ab579a98aeac"}
Sep 15 16:27:56.539864 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:56.539815    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ffhfm" podStartSLOduration=17.773789179 podStartE2EDuration="19.539798255s" totalImagesPullingTime="1.766009076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:27:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:56.539243463 +0000 UTC m=+64.907068652" watchObservedRunningTime="2026-09-15 16:27:56.539798255 +0000 UTC m=+64.907623454"
Sep 15 16:27:56.957327 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:56.957288    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:56.957511 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:56.957347    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:27:56.958230 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:56.958206    2566 scope.go:117] "RemoveContainer" containerID="b6b14dadd8c71c0d9548fa2262bf1196893452426c418fb8723e632c68830371"
Sep 15 16:27:56.958774 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:27:56.958736    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-s5lrt_openshift-console-operator(8f4bb97b-0349-4136-bb5e-2f4c089d367a)\"" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" podUID="8f4bb97b-0349-4136-bb5e-2f4c089d367a"
Sep 15 16:27:57.885976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:57.885938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:57.885976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:57.885983    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:57.888884 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:57.888865    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 15 16:27:57.889639 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:57.889623    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 15 16:27:57.898513 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:57.898492    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c3f47547-0e34-4911-a0e7-1f136f79d630-metrics-certs\") pod \"network-metrics-daemon-9t8nf\" (UID: \"c3f47547-0e34-4911-a0e7-1f136f79d630\") " pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:57.898660 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:57.898639    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8e9dce9b-8568-498c-a7f8-cfe8d55410c1-original-pull-secret\") pod \"global-pull-secret-syncer-zf92w\" (UID: \"8e9dce9b-8568-498c-a7f8-cfe8d55410c1\") " pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:57.986564 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:57.986534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:57.988803 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:57.988778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqvn4\" (UniqueName: \"kubernetes.io/projected/c9072b15-5772-442d-9666-6be71bffbe35-kube-api-access-zqvn4\") pod \"network-check-target-f9p97\" (UID: \"c9072b15-5772-442d-9666-6be71bffbe35\") " pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:58.067845 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.067819    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fhmxd\""
Sep 15 16:27:58.071963 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.071949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zf92w"
Sep 15 16:27:58.076093 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.076078    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:58.079453 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.079437    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkdt8\""
Sep 15 16:27:58.087746 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.087720    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9t8nf"
Sep 15 16:27:58.371820 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.371799    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zf92w"]
Sep 15 16:27:58.373946 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:58.373927    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e9dce9b_8568_498c_a7f8_cfe8d55410c1.slice/crio-91ef2373556370c2577c32feb4649169c05fa3f9e8d2a76beb5ce5da2ffb45a5 WatchSource:0}: Error finding container 91ef2373556370c2577c32feb4649169c05fa3f9e8d2a76beb5ce5da2ffb45a5: Status 404 returned error can't find the container with id 91ef2373556370c2577c32feb4649169c05fa3f9e8d2a76beb5ce5da2ffb45a5
Sep 15 16:27:58.391537 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.391516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:58.393835 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.393815    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"image-registry-d87d77784-67zfs\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") " pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:58.413039 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.412981    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-f9p97"]
Sep 15 16:27:58.416429 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:58.416409    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9072b15_5772_442d_9666_6be71bffbe35.slice/crio-51efd0b61730399f145c68b68e848aeab5f9910a802b912d1f4c401c5971694c WatchSource:0}: Error finding container 51efd0b61730399f145c68b68e848aeab5f9910a802b912d1f4c401c5971694c: Status 404 returned error can't find the container with id 51efd0b61730399f145c68b68e848aeab5f9910a802b912d1f4c401c5971694c
Sep 15 16:27:58.442495 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.442469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9t8nf"]
Sep 15 16:27:58.444769 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:58.444746    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f47547_0e34_4911_a0e7_1f136f79d630.slice/crio-bfa2164f2c0fca7feab1051af78e34507f0e23ee2c69c5249aa59557a562990f WatchSource:0}: Error finding container bfa2164f2c0fca7feab1051af78e34507f0e23ee2c69c5249aa59557a562990f: Status 404 returned error can't find the container with id bfa2164f2c0fca7feab1051af78e34507f0e23ee2c69c5249aa59557a562990f
Sep 15 16:27:58.492435 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.492411    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:58.492527 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.492451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:58.492527 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.492478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:58.492527 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.492515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:58.492659 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.492543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:58.493217 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.493191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6385b954-9257-40b7-a82b-d32c090cd40e-service-ca-bundle\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:58.494873 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.494844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6385b954-9257-40b7-a82b-d32c090cd40e-metrics-certs\") pod \"router-default-7f5d8767b4-qhwn7\" (UID: \"6385b954-9257-40b7-a82b-d32c090cd40e\") " pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:58.494873 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.494868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/63e81776-6e13-4425-b7e1-a5f42ea0ce43-metrics-tls\") pod \"dns-default-l9b4d\" (UID: \"63e81776-6e13-4425-b7e1-a5f42ea0ce43\") " pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:58.494992 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.494970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0e1009b7-c437-4e0f-801e-96e3106c9524-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-lrzv9\" (UID: \"0e1009b7-c437-4e0f-801e-96e3106c9524\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:58.495247 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.495228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/552d2031-4b9e-477f-aff2-09b7ba12f053-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-8gs62\" (UID: \"552d2031-4b9e-477f-aff2-09b7ba12f053\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:58.528776 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.528741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zf92w" event={"ID":"8e9dce9b-8568-498c-a7f8-cfe8d55410c1","Type":"ContainerStarted","Data":"91ef2373556370c2577c32feb4649169c05fa3f9e8d2a76beb5ce5da2ffb45a5"}
Sep 15 16:27:58.529806 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.529772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9t8nf" event={"ID":"c3f47547-0e34-4911-a0e7-1f136f79d630","Type":"ContainerStarted","Data":"bfa2164f2c0fca7feab1051af78e34507f0e23ee2c69c5249aa59557a562990f"}
Sep 15 16:27:58.530863 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.530844    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f9p97" event={"ID":"c9072b15-5772-442d-9666-6be71bffbe35","Type":"ContainerStarted","Data":"51efd0b61730399f145c68b68e848aeab5f9910a802b912d1f4c401c5971694c"}
Sep 15 16:27:58.609278 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.609249    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-whzsv\""
Sep 15 16:27:58.617033 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.617015    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:58.635302 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.635278    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mzxqz\""
Sep 15 16:27:58.643258 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.643240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:27:58.651994 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.651972    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6cgc5\""
Sep 15 16:27:58.660239 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.660213    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"
Sep 15 16:27:58.708795 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.708771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-x7vpq\""
Sep 15 16:27:58.717213 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.716976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"
Sep 15 16:27:58.718547 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.718521    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-njfkt\""
Sep 15 16:27:58.726184 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.726143    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:58.924823 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.924469    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d87d77784-67zfs"]
Sep 15 16:27:58.927265 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:58.927235    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df29ac7_55fb_42c0_bec2_f5375a8543ac.slice/crio-9e0b68fbccaca02ff5481c1f981dc8c1c78376d7b71f5e9773bebfc02f95eb8f WatchSource:0}: Error finding container 9e0b68fbccaca02ff5481c1f981dc8c1c78376d7b71f5e9773bebfc02f95eb8f: Status 404 returned error can't find the container with id 9e0b68fbccaca02ff5481c1f981dc8c1c78376d7b71f5e9773bebfc02f95eb8f
Sep 15 16:27:58.944815 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.944732    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l9b4d"]
Sep 15 16:27:58.947408 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:58.947375    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e81776_6e13_4425_b7e1_a5f42ea0ce43.slice/crio-70998e00bb73007444a1ccce99807b3042f747a807e57de1230b96b53497178b WatchSource:0}: Error finding container 70998e00bb73007444a1ccce99807b3042f747a807e57de1230b96b53497178b: Status 404 returned error can't find the container with id 70998e00bb73007444a1ccce99807b3042f747a807e57de1230b96b53497178b
Sep 15 16:27:58.977522 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:58.977491    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62"]
Sep 15 16:27:59.020741 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.020678    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9"]
Sep 15 16:27:59.026035 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:59.025999    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1009b7_c437_4e0f_801e_96e3106c9524.slice/crio-65ea93dcb13efd183ff818acfe2c1a93be9584121be7df205f734b2ba46f799b WatchSource:0}: Error finding container 65ea93dcb13efd183ff818acfe2c1a93be9584121be7df205f734b2ba46f799b: Status 404 returned error can't find the container with id 65ea93dcb13efd183ff818acfe2c1a93be9584121be7df205f734b2ba46f799b
Sep 15 16:27:59.045624 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.045599    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7f5d8767b4-qhwn7"]
Sep 15 16:27:59.049979 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:27:59.049949    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6385b954_9257_40b7_a82b_d32c090cd40e.slice/crio-9f03a37c52df97b2a3aa989646a5f8888c2bc713961adfb014e96746c39846bc WatchSource:0}: Error finding container 9f03a37c52df97b2a3aa989646a5f8888c2bc713961adfb014e96746c39846bc: Status 404 returned error can't find the container with id 9f03a37c52df97b2a3aa989646a5f8888c2bc713961adfb014e96746c39846bc
Sep 15 16:27:59.538700 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.537953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-f9p97" event={"ID":"c9072b15-5772-442d-9666-6be71bffbe35","Type":"ContainerStarted","Data":"ef6ea22e4329b6964a2ba0037b5b9dc097cb183aa62767b8c129694733065089"}
Sep 15 16:27:59.538700 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.538654    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:27:59.541107 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.541078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d87d77784-67zfs" event={"ID":"8df29ac7-55fb-42c0-bec2-f5375a8543ac","Type":"ContainerStarted","Data":"7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c"}
Sep 15 16:27:59.541261 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.541125    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:27:59.541261 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.541147    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d87d77784-67zfs" event={"ID":"8df29ac7-55fb-42c0-bec2-f5375a8543ac","Type":"ContainerStarted","Data":"9e0b68fbccaca02ff5481c1f981dc8c1c78376d7b71f5e9773bebfc02f95eb8f"}
Sep 15 16:27:59.543035 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.543001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7" event={"ID":"6385b954-9257-40b7-a82b-d32c090cd40e","Type":"ContainerStarted","Data":"adbad6e2fbcab5758209057dace1f234d784dba2a7e5445896688a3a167ba94e"}
Sep 15 16:27:59.543141 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.543041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7" event={"ID":"6385b954-9257-40b7-a82b-d32c090cd40e","Type":"ContainerStarted","Data":"9f03a37c52df97b2a3aa989646a5f8888c2bc713961adfb014e96746c39846bc"}
Sep 15 16:27:59.544319 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.544295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9" event={"ID":"0e1009b7-c437-4e0f-801e-96e3106c9524","Type":"ContainerStarted","Data":"65ea93dcb13efd183ff818acfe2c1a93be9584121be7df205f734b2ba46f799b"}
Sep 15 16:27:59.545459 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.545435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9b4d" event={"ID":"63e81776-6e13-4425-b7e1-a5f42ea0ce43","Type":"ContainerStarted","Data":"70998e00bb73007444a1ccce99807b3042f747a807e57de1230b96b53497178b"}
Sep 15 16:27:59.546795 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.546767    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62" event={"ID":"552d2031-4b9e-477f-aff2-09b7ba12f053","Type":"ContainerStarted","Data":"7a26f08dec2ba6caf86e75e27bc21f1ecc8d7c00ccc0f2da96206403150e8327"}
Sep 15 16:27:59.572076 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.572019    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-f9p97" podStartSLOduration=66.571997589 podStartE2EDuration="1m6.571997589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:59.569878485 +0000 UTC m=+67.937703685" watchObservedRunningTime="2026-09-15 16:27:59.571997589 +0000 UTC m=+67.939822789"
Sep 15 16:27:59.603663 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.603432    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7" podStartSLOduration=36.603412555 podStartE2EDuration="36.603412555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:59.602977669 +0000 UTC m=+67.970802869" watchObservedRunningTime="2026-09-15 16:27:59.603412555 +0000 UTC m=+67.971237755"
Sep 15 16:27:59.727062 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.727020    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:59.729830 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.729655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:27:59.752684 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:27:59.752619    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d87d77784-67zfs" podStartSLOduration=66.752595992 podStartE2EDuration="1m6.752595992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:27:59.622876338 +0000 UTC m=+67.990701555" watchObservedRunningTime="2026-09-15 16:27:59.752595992 +0000 UTC m=+68.120421204"
Sep 15 16:28:00.556424 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:00.555434    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9t8nf" event={"ID":"c3f47547-0e34-4911-a0e7-1f136f79d630","Type":"ContainerStarted","Data":"6d15df584066e10bca13fb13b039006487981ca1bb656becec272d3dd12202ed"}
Sep 15 16:28:00.556424 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:00.555497    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9t8nf" event={"ID":"c3f47547-0e34-4911-a0e7-1f136f79d630","Type":"ContainerStarted","Data":"6d9664770f5b5fec410ce0d71411e7d6d0f7d07d8c0f01cdd7428f378d82c74a"}
Sep 15 16:28:00.556424 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:00.556259    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:28:00.558024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:00.557999    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7f5d8767b4-qhwn7"
Sep 15 16:28:00.575662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:00.575355    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9t8nf" podStartSLOduration=66.492713608 podStartE2EDuration="1m7.575334426s" totalImagesPullingTime="1.082620818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:00.5730895 +0000 UTC m=+68.940914699" watchObservedRunningTime="2026-09-15 16:28:00.575334426 +0000 UTC m=+68.943159643"
Sep 15 16:28:03.569692 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:03.569641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9" event={"ID":"0e1009b7-c437-4e0f-801e-96e3106c9524","Type":"ContainerStarted","Data":"30e14252acdbf68671d6ef9a3dd318f75cd03f62e8434dbbf6b0fdbe4b05ccef"}
Sep 15 16:28:03.572903 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:03.572268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9b4d" event={"ID":"63e81776-6e13-4425-b7e1-a5f42ea0ce43","Type":"ContainerStarted","Data":"677c580f9254ef4b205b38952623ab9eb39ff76c0f0e593fd492fc33c14e5356"}
Sep 15 16:28:03.575182 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:03.575016    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62" event={"ID":"552d2031-4b9e-477f-aff2-09b7ba12f053","Type":"ContainerStarted","Data":"e81db7fe0072ac23bf6715218df785e4ce2f222ec85e705382909966f84ed4ed"}
Sep 15 16:28:03.588165 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:03.588111    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-lrzv9" podStartSLOduration=36.163375227 podStartE2EDuration="40.588089802s" totalImagesPullingTime="4.424714575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:03.586551882 +0000 UTC m=+71.954377082" watchObservedRunningTime="2026-09-15 16:28:03.588089802 +0000 UTC m=+71.955915003"
Sep 15 16:28:04.580229 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:04.580192    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l9b4d" event={"ID":"63e81776-6e13-4425-b7e1-a5f42ea0ce43","Type":"ContainerStarted","Data":"bc85d4074b6abb9fd446edb72e6d8d2455ce33296ca74f999913d8f469f9d5c3"}
Sep 15 16:28:04.580719 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:04.580395    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:28:04.581798 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:04.581779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zf92w" event={"ID":"8e9dce9b-8568-498c-a7f8-cfe8d55410c1","Type":"ContainerStarted","Data":"01351cb7158bda181ea9f28e4f56c0842b0fec4b5dbacbc67e5ccb03805457c3"}
Sep 15 16:28:04.583426 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:04.583404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62" event={"ID":"552d2031-4b9e-477f-aff2-09b7ba12f053","Type":"ContainerStarted","Data":"a63fd8d9d4d5b912d5adcc665c9d1e36a6727b64490dc0a95c64443206cde52f"}
Sep 15 16:28:04.600486 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:04.600434    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l9b4d" podStartSLOduration=34.096543075 podStartE2EDuration="38.600421347s" totalImagesPullingTime="4.503878272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:04.59955031 +0000 UTC m=+72.967375536" watchObservedRunningTime="2026-09-15 16:28:04.600421347 +0000 UTC m=+72.968246546"
Sep 15 16:28:04.615333 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:04.615290    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-8gs62" podStartSLOduration=37.189721531 podStartE2EDuration="41.615276913s" totalImagesPullingTime="4.425555382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:04.614704883 +0000 UTC m=+72.982530082" watchObservedRunningTime="2026-09-15 16:28:04.615276913 +0000 UTC m=+72.983102112"
Sep 15 16:28:04.629340 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:04.629296    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zf92w" podStartSLOduration=66.538898256 podStartE2EDuration="1m11.629280839s" totalImagesPullingTime="5.090382583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 16:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:04.628314289 +0000 UTC m=+72.996139490" watchObservedRunningTime="2026-09-15 16:28:04.629280839 +0000 UTC m=+72.997106086"
Sep 15 16:28:05.137636 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.137591    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"]
Sep 15 16:28:05.172887 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.172858    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"]
Sep 15 16:28:05.173021 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.172963    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.176389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.176368    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 15 16:28:05.176511 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.176373    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 15 16:28:05.176511 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.176411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 15 16:28:05.176681 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.176664    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 15 16:28:05.244382 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.244353    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"]
Sep 15 16:28:05.268508 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.268475    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"
Sep 15 16:28:05.273703 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.273672    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 15 16:28:05.274111 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.274096    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-kjxh2\""
Sep 15 16:28:05.297507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.297484    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"]
Sep 15 16:28:05.347152 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.347123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-tmp\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.347152 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.347154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5ff9370f-5788-4cad-9d0a-7c324165119f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7mxvm\" (UID: \"5ff9370f-5788-4cad-9d0a-7c324165119f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"
Sep 15 16:28:05.347311 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.347203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.347311 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.347242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzwg5\" (UniqueName: \"kubernetes.io/projected/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-kube-api-access-bzwg5\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.448473 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.448383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-tmp\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.448473 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.448418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5ff9370f-5788-4cad-9d0a-7c324165119f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7mxvm\" (UID: \"5ff9370f-5788-4cad-9d0a-7c324165119f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"
Sep 15 16:28:05.448692 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.448556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.448692 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.448671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzwg5\" (UniqueName: \"kubernetes.io/projected/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-kube-api-access-bzwg5\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.448895 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.448862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-tmp\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.451035 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.451010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5ff9370f-5788-4cad-9d0a-7c324165119f-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-7mxvm\" (UID: \"5ff9370f-5788-4cad-9d0a-7c324165119f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"
Sep 15 16:28:05.451147 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.451064    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-klusterlet-config\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.459120 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.459093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzwg5\" (UniqueName: \"kubernetes.io/projected/49e49dbd-dc7a-49b1-81e2-a9ec427f931d-kube-api-access-bzwg5\") pod \"klusterlet-addon-workmgr-84f446b458-glqtz\" (UID: \"49e49dbd-dc7a-49b1-81e2-a9ec427f931d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.482867 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.482837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:05.577694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.577665    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"
Sep 15 16:28:05.783543 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.783473    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"]
Sep 15 16:28:05.786875 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:28:05.786850    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49e49dbd_dc7a_49b1_81e2_a9ec427f931d.slice/crio-09c1f9509abb752b9c2f6f636126b5b05e3312f897fc00d94ee9d19a8b9c6242 WatchSource:0}: Error finding container 09c1f9509abb752b9c2f6f636126b5b05e3312f897fc00d94ee9d19a8b9c6242: Status 404 returned error can't find the container with id 09c1f9509abb752b9c2f6f636126b5b05e3312f897fc00d94ee9d19a8b9c6242
Sep 15 16:28:05.875646 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:05.875511    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"]
Sep 15 16:28:05.878187 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:28:05.878157    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff9370f_5788_4cad_9d0a_7c324165119f.slice/crio-8143b8591c5b7fdb987d85987460efd89cc1cb3defa8718ff39bd87723afe24f WatchSource:0}: Error finding container 8143b8591c5b7fdb987d85987460efd89cc1cb3defa8718ff39bd87723afe24f: Status 404 returned error can't find the container with id 8143b8591c5b7fdb987d85987460efd89cc1cb3defa8718ff39bd87723afe24f
Sep 15 16:28:06.596932 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:06.596848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz" event={"ID":"49e49dbd-dc7a-49b1-81e2-a9ec427f931d","Type":"ContainerStarted","Data":"09c1f9509abb752b9c2f6f636126b5b05e3312f897fc00d94ee9d19a8b9c6242"}
Sep 15 16:28:06.598478 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:06.598448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm" event={"ID":"5ff9370f-5788-4cad-9d0a-7c324165119f","Type":"ContainerStarted","Data":"8143b8591c5b7fdb987d85987460efd89cc1cb3defa8718ff39bd87723afe24f"}
Sep 15 16:28:09.153731 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:09.153682    2566 scope.go:117] "RemoveContainer" containerID="b6b14dadd8c71c0d9548fa2262bf1196893452426c418fb8723e632c68830371"
Sep 15 16:28:09.612958 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:09.612938    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:28:09.613052 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:09.613032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" event={"ID":"8f4bb97b-0349-4136-bb5e-2f4c089d367a","Type":"ContainerStarted","Data":"d3e39f57f6249a65cd4ef1bc017d8f7820cc7757c18d8cb460dd04f0edba1603"}
Sep 15 16:28:10.617915 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.617875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm" event={"ID":"5ff9370f-5788-4cad-9d0a-7c324165119f","Type":"ContainerStarted","Data":"317602acbdcd58c13ff2a43a6072f6527d5ef4b141f4c7146734c725c65b7af6"}
Sep 15 16:28:10.618380 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.618046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"
Sep 15 16:28:10.619932 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.619603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz" event={"ID":"49e49dbd-dc7a-49b1-81e2-a9ec427f931d","Type":"ContainerStarted","Data":"fd521882048345b5d00868311f77bd2c2013392ba570b9ef1c1b2ef23f1b7d79"}
Sep 15 16:28:10.619932 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.619740    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:10.619932 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.619876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:28:10.621177 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.621160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz"
Sep 15 16:28:10.623376 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.623356    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm"
Sep 15 16:28:10.647921 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.647873    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-7mxvm" podStartSLOduration=2.002383929 podStartE2EDuration="5.64785509s" totalImagesPullingTime="3.645471161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:10.64785473 +0000 UTC m=+79.015679929" watchObservedRunningTime="2026-09-15 16:28:10.64785509 +0000 UTC m=+79.015680293"
Sep 15 16:28:10.667006 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.666963    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt" podStartSLOduration=38.900680811 podStartE2EDuration="46.66695032s" totalImagesPullingTime="7.766269509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:10.665929701 +0000 UTC m=+79.033754901" watchObservedRunningTime="2026-09-15 16:28:10.66695032 +0000 UTC m=+79.034775520"
Sep 15 16:28:10.681553 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:10.681511    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84f446b458-glqtz" podStartSLOduration=1.5143258670000002 podStartE2EDuration="5.681498941s" totalImagesPullingTime="4.167173074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:10.681134683 +0000 UTC m=+79.048959884" watchObservedRunningTime="2026-09-15 16:28:10.681498941 +0000 UTC m=+79.049324141"
Sep 15 16:28:11.032436 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:11.032406    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-s5lrt"
Sep 15 16:28:14.590355 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:14.590248    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l9b4d"
Sep 15 16:28:15.532319 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.532278    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vjrc8"]
Sep 15 16:28:15.535281 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.535257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.537981 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.537960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 15 16:28:15.538098 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.537982    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-9ttpl\""
Sep 15 16:28:15.538148 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.538127    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 15 16:28:15.538264 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.538248    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 15 16:28:15.538846 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.538832    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 15 16:28:15.635962 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.635931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.636341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.635971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56262877-35f7-49ee-a216-5d548ec30699-metrics-client-ca\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.636341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.635997    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-sys\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.636341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.636040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-root\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.636341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.636076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-tls\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.636341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.636099    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.636341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.636172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-textfile\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.636341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.636205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-wtmp\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.636341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.636222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvxqt\" (UniqueName: \"kubernetes.io/projected/56262877-35f7-49ee-a216-5d548ec30699-kube-api-access-nvxqt\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737449 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-sys\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737449 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737451    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-root\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737670 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737475    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-tls\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737670 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-sys\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737670 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737540    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-root\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737670 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737816 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737674    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-textfile\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737816 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-wtmp\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737816 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvxqt\" (UniqueName: \"kubernetes.io/projected/56262877-35f7-49ee-a216-5d548ec30699-kube-api-access-nvxqt\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737948 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.737948 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737875    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56262877-35f7-49ee-a216-5d548ec30699-metrics-client-ca\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.738039 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-wtmp\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.738039 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.737970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-textfile\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.738250 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.738230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.738527 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.738510    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/56262877-35f7-49ee-a216-5d548ec30699-metrics-client-ca\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.740021 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.740004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-tls\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.740133 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.740115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/56262877-35f7-49ee-a216-5d548ec30699-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.746379 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.746359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvxqt\" (UniqueName: \"kubernetes.io/projected/56262877-35f7-49ee-a216-5d548ec30699-kube-api-access-nvxqt\") pod \"node-exporter-vjrc8\" (UID: \"56262877-35f7-49ee-a216-5d548ec30699\") " pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.844449 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:15.844370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjrc8"
Sep 15 16:28:15.855152 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:28:15.855124    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56262877_35f7_49ee_a216_5d548ec30699.slice/crio-977eda0a3f0d2b21e47de91e46154fe484c57f96d2da0636112d9042653b5d3f WatchSource:0}: Error finding container 977eda0a3f0d2b21e47de91e46154fe484c57f96d2da0636112d9042653b5d3f: Status 404 returned error can't find the container with id 977eda0a3f0d2b21e47de91e46154fe484c57f96d2da0636112d9042653b5d3f
Sep 15 16:28:16.644407 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:16.644370    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjrc8" event={"ID":"56262877-35f7-49ee-a216-5d548ec30699","Type":"ContainerStarted","Data":"977eda0a3f0d2b21e47de91e46154fe484c57f96d2da0636112d9042653b5d3f"}
Sep 15 16:28:17.650343 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:17.650312    2566 generic.go:356] "Generic (PLEG): container finished" podID="56262877-35f7-49ee-a216-5d548ec30699" containerID="ca2911b131a12a77eab3e9b4d1f6495044f93cf09f767537938fcdbaca8cf42c" exitCode=0
Sep 15 16:28:17.650745 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:17.650402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjrc8" event={"ID":"56262877-35f7-49ee-a216-5d548ec30699","Type":"ContainerDied","Data":"ca2911b131a12a77eab3e9b4d1f6495044f93cf09f767537938fcdbaca8cf42c"}
Sep 15 16:28:18.621916 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:18.621871    2566 patch_prober.go:28] interesting pod/image-registry-d87d77784-67zfs container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 15 16:28:18.622293 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:18.621944    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d87d77784-67zfs" podUID="8df29ac7-55fb-42c0-bec2-f5375a8543ac" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 15 16:28:18.656315 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:18.656282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjrc8" event={"ID":"56262877-35f7-49ee-a216-5d548ec30699","Type":"ContainerStarted","Data":"6218272ea4189885eaf55996fe10938a1025b80d870369dce8a54ea48277f1ca"}
Sep 15 16:28:18.656689 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:18.656334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjrc8" event={"ID":"56262877-35f7-49ee-a216-5d548ec30699","Type":"ContainerStarted","Data":"956dacbf2d62a4fa899556f8e162929294c66e54d36d81f54f8eec5ccfbced92"}
Sep 15 16:28:18.679148 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:18.679097    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vjrc8" podStartSLOduration=2.297982731 podStartE2EDuration="3.679082153s" totalImagesPullingTime="1.315366326s" totalInitContainerRuntime="65.733096ms" isStatefulPod=true podCreationTimestamp="2026-09-15 16:28:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:18.67717163 +0000 UTC m=+87.044996840" watchObservedRunningTime="2026-09-15 16:28:18.679082153 +0000 UTC m=+87.046907352"
Sep 15 16:28:20.560753 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:20.560724    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:28:28.516211 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:28.516177    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d87d77784-67zfs"]
Sep 15 16:28:31.121071 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.120978    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55859b45fc-tjnwx"]
Sep 15 16:28:31.124034 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.124009    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.128168 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.127929    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kgwqw\""
Sep 15 16:28:31.128168 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.127973    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 15 16:28:31.128168 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.127991    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 15 16:28:31.128168 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.127929    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 15 16:28:31.128168 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.128037    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 15 16:28:31.128168 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.128058    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 15 16:28:31.128493 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.128242    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 15 16:28:31.128493 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.128357    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 15 16:28:31.132970 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.132946    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 15 16:28:31.136760 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.136741    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55859b45fc-tjnwx"]
Sep 15 16:28:31.156809 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.156786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-oauth-serving-cert\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.156930 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.156820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-service-ca\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.156930 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.156852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mkh\" (UniqueName: \"kubernetes.io/projected/90657cd0-e12f-4ebe-90af-f63fee3e3105-kube-api-access-49mkh\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.157018 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.156969    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-serving-cert\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.157018 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.156992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-trusted-ca-bundle\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.157018 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.157013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-config\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.157134 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.157082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-oauth-config\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.257683 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.257649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-service-ca\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.257683 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.257687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mkh\" (UniqueName: \"kubernetes.io/projected/90657cd0-e12f-4ebe-90af-f63fee3e3105-kube-api-access-49mkh\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.257887 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.257745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-serving-cert\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.257887 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.257768    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-trusted-ca-bundle\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.257887 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.257793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-config\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.257887 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.257835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-oauth-config\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.258079 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.257965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-oauth-serving-cert\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.258466 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.258442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-service-ca\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.258671 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.258650    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-config\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.258671 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.258665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-trusted-ca-bundle\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.258780 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.258678    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-oauth-serving-cert\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.260745 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.260721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-serving-cert\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.260845 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.260825    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-oauth-config\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.265485 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.265459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mkh\" (UniqueName: \"kubernetes.io/projected/90657cd0-e12f-4ebe-90af-f63fee3e3105-kube-api-access-49mkh\") pod \"console-55859b45fc-tjnwx\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") " pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.434149 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.434112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:31.561271 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.561240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-f9p97"
Sep 15 16:28:31.713862 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:31.713792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55859b45fc-tjnwx"]
Sep 15 16:28:31.716726 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:28:31.716693    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90657cd0_e12f_4ebe_90af_f63fee3e3105.slice/crio-44273ea7c84b5c0cd480349a7bb14907958b068c225861ce93236b063ce0a374 WatchSource:0}: Error finding container 44273ea7c84b5c0cd480349a7bb14907958b068c225861ce93236b063ce0a374: Status 404 returned error can't find the container with id 44273ea7c84b5c0cd480349a7bb14907958b068c225861ce93236b063ce0a374
Sep 15 16:28:32.708587 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:32.708540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55859b45fc-tjnwx" event={"ID":"90657cd0-e12f-4ebe-90af-f63fee3e3105","Type":"ContainerStarted","Data":"44273ea7c84b5c0cd480349a7bb14907958b068c225861ce93236b063ce0a374"}
Sep 15 16:28:35.721249 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:35.721204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55859b45fc-tjnwx" event={"ID":"90657cd0-e12f-4ebe-90af-f63fee3e3105","Type":"ContainerStarted","Data":"516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5"}
Sep 15 16:28:35.741939 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:35.741868    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55859b45fc-tjnwx" podStartSLOduration=1.671247103 podStartE2EDuration="4.741854294s" totalImagesPullingTime="3.070607191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:35.740459349 +0000 UTC m=+104.108284549" watchObservedRunningTime="2026-09-15 16:28:35.741854294 +0000 UTC m=+104.109679493"
Sep 15 16:28:38.823002 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.822967    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-647d99ddbf-n8525"]
Sep 15 16:28:38.825716 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.825698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:38.837643 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.837617    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647d99ddbf-n8525"]
Sep 15 16:28:38.923065 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.923038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-oauth-serving-cert\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:38.923226 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.923080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-serving-cert\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:38.923226 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.923100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk7x8\" (UniqueName: \"kubernetes.io/projected/c113c6c4-8457-4a77-825a-c879a6fe75b2-kube-api-access-vk7x8\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:38.923226 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.923189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-oauth-config\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:38.923348 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.923244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-config\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:38.923348 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.923289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-service-ca\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:38.923348 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:38.923315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-trusted-ca-bundle\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.023966 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.023937    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-oauth-serving-cert\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024108 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.023977    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-serving-cert\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024108 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.023994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk7x8\" (UniqueName: \"kubernetes.io/projected/c113c6c4-8457-4a77-825a-c879a6fe75b2-kube-api-access-vk7x8\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024188 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.024107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-oauth-config\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024188 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.024177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-config\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024317 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.024204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-service-ca\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024317 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.024235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-trusted-ca-bundle\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024755 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.024725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-oauth-serving-cert\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024870 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.024850    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-service-ca\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024870 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.024863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-config\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.024978 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.024953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-trusted-ca-bundle\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.026501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.026460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-oauth-config\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.026619 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.026564    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-serving-cert\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.032415 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.032394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk7x8\" (UniqueName: \"kubernetes.io/projected/c113c6c4-8457-4a77-825a-c879a6fe75b2-kube-api-access-vk7x8\") pod \"console-647d99ddbf-n8525\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") " pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.135022 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.134989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:39.405003 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.404978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-647d99ddbf-n8525"]
Sep 15 16:28:39.407238 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:28:39.407210    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc113c6c4_8457_4a77_825a_c879a6fe75b2.slice/crio-e8c942d220d480c8db452f4ce7fddb2c4a5bc5181d2ff828013bec7c655f90fc WatchSource:0}: Error finding container e8c942d220d480c8db452f4ce7fddb2c4a5bc5181d2ff828013bec7c655f90fc: Status 404 returned error can't find the container with id e8c942d220d480c8db452f4ce7fddb2c4a5bc5181d2ff828013bec7c655f90fc
Sep 15 16:28:39.736482 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.736401    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647d99ddbf-n8525" event={"ID":"c113c6c4-8457-4a77-825a-c879a6fe75b2","Type":"ContainerStarted","Data":"ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485"}
Sep 15 16:28:39.736482 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.736449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647d99ddbf-n8525" event={"ID":"c113c6c4-8457-4a77-825a-c879a6fe75b2","Type":"ContainerStarted","Data":"e8c942d220d480c8db452f4ce7fddb2c4a5bc5181d2ff828013bec7c655f90fc"}
Sep 15 16:28:39.757161 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:39.757112    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-647d99ddbf-n8525" podStartSLOduration=1.757094599 podStartE2EDuration="1.757094599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:28:39.7555531 +0000 UTC m=+108.123378300" watchObservedRunningTime="2026-09-15 16:28:39.757094599 +0000 UTC m=+108.124919797"
Sep 15 16:28:41.434895 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:41.434864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:41.435284 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:41.435136    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:41.444269 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:41.440391    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:41.749138 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:41.749063    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:28:49.135756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:49.135711    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:49.136452 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:49.135786    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:49.140605 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:49.140562    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:49.777758 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:49.777730    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:28:49.827890 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:49.827846    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55859b45fc-tjnwx"]
Sep 15 16:28:51.785488 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:51.785394    2566 generic.go:356] "Generic (PLEG): container finished" podID="537e42cc-e1d6-4cb1-80d4-fd0209023ec9" containerID="904875b9406bcd0b849692b56993a9f857524e286ac313346b54749bb3b7cee6" exitCode=0
Sep 15 16:28:51.785488 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:51.785469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht" event={"ID":"537e42cc-e1d6-4cb1-80d4-fd0209023ec9","Type":"ContainerDied","Data":"904875b9406bcd0b849692b56993a9f857524e286ac313346b54749bb3b7cee6"}
Sep 15 16:28:51.785913 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:51.785870    2566 scope.go:117] "RemoveContainer" containerID="904875b9406bcd0b849692b56993a9f857524e286ac313346b54749bb3b7cee6"
Sep 15 16:28:52.790935 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:52.790901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-n8wht" event={"ID":"537e42cc-e1d6-4cb1-80d4-fd0209023ec9","Type":"ContainerStarted","Data":"d1e4c8797ee00c04104559e0831eb917f921f48d7607fb1cbb69d465f558f68a"}
Sep 15 16:28:53.537623 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.537565    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d87d77784-67zfs" podUID="8df29ac7-55fb-42c0-bec2-f5375a8543ac" containerName="registry" containerID="cri-o://7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c" gracePeriod=30
Sep 15 16:28:53.777365 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.777339    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:28:53.795151 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.795084    2566 generic.go:356] "Generic (PLEG): container finished" podID="8df29ac7-55fb-42c0-bec2-f5375a8543ac" containerID="7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c" exitCode=0
Sep 15 16:28:53.795151 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.795136    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d87d77784-67zfs"
Sep 15 16:28:53.795496 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.795168    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d87d77784-67zfs" event={"ID":"8df29ac7-55fb-42c0-bec2-f5375a8543ac","Type":"ContainerDied","Data":"7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c"}
Sep 15 16:28:53.795496 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.795214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d87d77784-67zfs" event={"ID":"8df29ac7-55fb-42c0-bec2-f5375a8543ac","Type":"ContainerDied","Data":"9e0b68fbccaca02ff5481c1f981dc8c1c78376d7b71f5e9773bebfc02f95eb8f"}
Sep 15 16:28:53.795496 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.795233    2566 scope.go:117] "RemoveContainer" containerID="7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c"
Sep 15 16:28:53.804852 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.804481    2566 scope.go:117] "RemoveContainer" containerID="7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c"
Sep 15 16:28:53.810254 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:28:53.810221    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c\": container with ID starting with 7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c not found: ID does not exist" containerID="7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c"
Sep 15 16:28:53.810395 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.810261    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c"} err="failed to get container status \"7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c\": rpc error: code = NotFound desc = could not find container \"7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c\": container with ID starting with 7dde4ce24766f3a8d5c3c8559c9cfde3cf28f7b2943a649ac2ccb83e00bf430c not found: ID does not exist"
Sep 15 16:28:53.849144 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849116    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") pod \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") "
Sep 15 16:28:53.849254 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849162    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8df29ac7-55fb-42c0-bec2-f5375a8543ac-ca-trust-extracted\") pod \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") "
Sep 15 16:28:53.849254 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-installation-pull-secrets\") pod \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") "
Sep 15 16:28:53.849254 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849211    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-trusted-ca\") pod \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") "
Sep 15 16:28:53.849254 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-bound-sa-token\") pod \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") "
Sep 15 16:28:53.849254 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849250    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-certificates\") pod \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") "
Sep 15 16:28:53.849483 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-image-registry-private-configuration\") pod \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") "
Sep 15 16:28:53.849483 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrg28\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-kube-api-access-hrg28\") pod \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\" (UID: \"8df29ac7-55fb-42c0-bec2-f5375a8543ac\") "
Sep 15 16:28:53.849781 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849727    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8df29ac7-55fb-42c0-bec2-f5375a8543ac" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:28:53.849781 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.849763    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8df29ac7-55fb-42c0-bec2-f5375a8543ac" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:28:53.851684 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.851611    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8df29ac7-55fb-42c0-bec2-f5375a8543ac" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:28:53.851927 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.851888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8df29ac7-55fb-42c0-bec2-f5375a8543ac" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:28:53.852023 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.851945    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8df29ac7-55fb-42c0-bec2-f5375a8543ac" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:28:53.852023 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.851998    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8df29ac7-55fb-42c0-bec2-f5375a8543ac" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:28:53.852174 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.852153    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-kube-api-access-hrg28" (OuterVolumeSpecName: "kube-api-access-hrg28") pod "8df29ac7-55fb-42c0-bec2-f5375a8543ac" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac"). InnerVolumeSpecName "kube-api-access-hrg28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:28:53.857769 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.857746    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8df29ac7-55fb-42c0-bec2-f5375a8543ac-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8df29ac7-55fb-42c0-bec2-f5375a8543ac" (UID: "8df29ac7-55fb-42c0-bec2-f5375a8543ac"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:28:53.950617 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.950588    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:28:53.950617 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.950614    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8df29ac7-55fb-42c0-bec2-f5375a8543ac-ca-trust-extracted\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:28:53.950617 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.950624    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-installation-pull-secrets\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:28:53.950810 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.950633    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-trusted-ca\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:28:53.950810 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.950641    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-bound-sa-token\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:28:53.950810 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.950648    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8df29ac7-55fb-42c0-bec2-f5375a8543ac-registry-certificates\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:28:53.950810 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.950658    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8df29ac7-55fb-42c0-bec2-f5375a8543ac-image-registry-private-configuration\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:28:53.950810 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:53.950667    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrg28\" (UniqueName: \"kubernetes.io/projected/8df29ac7-55fb-42c0-bec2-f5375a8543ac-kube-api-access-hrg28\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:28:54.116350 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:54.116319    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d87d77784-67zfs"]
Sep 15 16:28:54.120781 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:54.120757    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d87d77784-67zfs"]
Sep 15 16:28:54.158184 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:28:54.158155    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8df29ac7-55fb-42c0-bec2-f5375a8543ac" path="/var/lib/kubelet/pods/8df29ac7-55fb-42c0-bec2-f5375a8543ac/volumes"
Sep 15 16:29:01.827694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:01.827657    2566 generic.go:356] "Generic (PLEG): container finished" podID="8f85e488-5160-4a79-a4b9-cf5c3c7734b5" containerID="3712400b416411186f234d7c02d0b2c4283c1410ad1fb55d2d5e7c05aa802a7e" exitCode=0
Sep 15 16:29:01.828095 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:01.827719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-nhpth" event={"ID":"8f85e488-5160-4a79-a4b9-cf5c3c7734b5","Type":"ContainerDied","Data":"3712400b416411186f234d7c02d0b2c4283c1410ad1fb55d2d5e7c05aa802a7e"}
Sep 15 16:29:01.828095 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:01.828036    2566 scope.go:117] "RemoveContainer" containerID="3712400b416411186f234d7c02d0b2c4283c1410ad1fb55d2d5e7c05aa802a7e"
Sep 15 16:29:02.833432 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:02.833395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-nhpth" event={"ID":"8f85e488-5160-4a79-a4b9-cf5c3c7734b5","Type":"ContainerStarted","Data":"92cb319292dfdc6c778bab76e57759894ff7034e8a50ccd957524f243b9a3d4a"}
Sep 15 16:29:06.849095 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:06.849055    2566 generic.go:356] "Generic (PLEG): container finished" podID="75fc8f59-26b4-4890-aa40-0ca7c847967f" containerID="b5d1c1ae592f0381d9c5b90504670aa72511918b870e0be57a79bd4c402fd5be" exitCode=0
Sep 15 16:29:06.849635 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:06.849127    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn" event={"ID":"75fc8f59-26b4-4890-aa40-0ca7c847967f","Type":"ContainerDied","Data":"b5d1c1ae592f0381d9c5b90504670aa72511918b870e0be57a79bd4c402fd5be"}
Sep 15 16:29:06.849635 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:06.849427    2566 scope.go:117] "RemoveContainer" containerID="b5d1c1ae592f0381d9c5b90504670aa72511918b870e0be57a79bd4c402fd5be"
Sep 15 16:29:07.854995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:07.854960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-zj6qn" event={"ID":"75fc8f59-26b4-4890-aa40-0ca7c847967f","Type":"ContainerStarted","Data":"4ff4ac768700950b3a5e9ccb6926cf36968519bb1c08be337574b04644ad4bcf"}
Sep 15 16:29:14.853481 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:14.853387    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55859b45fc-tjnwx" podUID="90657cd0-e12f-4ebe-90af-f63fee3e3105" containerName="console" containerID="cri-o://516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5" gracePeriod=15
Sep 15 16:29:15.123110 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.123079    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:29:15.220614 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.220552    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-config\") pod \"90657cd0-e12f-4ebe-90af-f63fee3e3105\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") "
Sep 15 16:29:15.220804 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.220671    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-oauth-serving-cert\") pod \"90657cd0-e12f-4ebe-90af-f63fee3e3105\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") "
Sep 15 16:29:15.220804 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.220703    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-serving-cert\") pod \"90657cd0-e12f-4ebe-90af-f63fee3e3105\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") "
Sep 15 16:29:15.220804 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.220734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-service-ca\") pod \"90657cd0-e12f-4ebe-90af-f63fee3e3105\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") "
Sep 15 16:29:15.220804 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.220788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49mkh\" (UniqueName: \"kubernetes.io/projected/90657cd0-e12f-4ebe-90af-f63fee3e3105-kube-api-access-49mkh\") pod \"90657cd0-e12f-4ebe-90af-f63fee3e3105\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") "
Sep 15 16:29:15.221015 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.220817    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-trusted-ca-bundle\") pod \"90657cd0-e12f-4ebe-90af-f63fee3e3105\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") "
Sep 15 16:29:15.221015 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.220848    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-oauth-config\") pod \"90657cd0-e12f-4ebe-90af-f63fee3e3105\" (UID: \"90657cd0-e12f-4ebe-90af-f63fee3e3105\") "
Sep 15 16:29:15.221015 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.220913    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-config" (OuterVolumeSpecName: "console-config") pod "90657cd0-e12f-4ebe-90af-f63fee3e3105" (UID: "90657cd0-e12f-4ebe-90af-f63fee3e3105"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:15.221171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.221045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "90657cd0-e12f-4ebe-90af-f63fee3e3105" (UID: "90657cd0-e12f-4ebe-90af-f63fee3e3105"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:15.221171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.221094    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-service-ca" (OuterVolumeSpecName: "service-ca") pod "90657cd0-e12f-4ebe-90af-f63fee3e3105" (UID: "90657cd0-e12f-4ebe-90af-f63fee3e3105"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:15.221171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.221133    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:29:15.221171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.221145    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-oauth-serving-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:29:15.221434 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.221328    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "90657cd0-e12f-4ebe-90af-f63fee3e3105" (UID: "90657cd0-e12f-4ebe-90af-f63fee3e3105"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:29:15.223281 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.223250    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "90657cd0-e12f-4ebe-90af-f63fee3e3105" (UID: "90657cd0-e12f-4ebe-90af-f63fee3e3105"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:15.223281 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.223267    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "90657cd0-e12f-4ebe-90af-f63fee3e3105" (UID: "90657cd0-e12f-4ebe-90af-f63fee3e3105"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:29:15.223410 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.223335    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90657cd0-e12f-4ebe-90af-f63fee3e3105-kube-api-access-49mkh" (OuterVolumeSpecName: "kube-api-access-49mkh") pod "90657cd0-e12f-4ebe-90af-f63fee3e3105" (UID: "90657cd0-e12f-4ebe-90af-f63fee3e3105"). InnerVolumeSpecName "kube-api-access-49mkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:29:15.321990 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.321958    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-serving-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:29:15.321990 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.321988    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-service-ca\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:29:15.322171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.321999    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49mkh\" (UniqueName: \"kubernetes.io/projected/90657cd0-e12f-4ebe-90af-f63fee3e3105-kube-api-access-49mkh\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:29:15.322171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.322010    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90657cd0-e12f-4ebe-90af-f63fee3e3105-trusted-ca-bundle\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:29:15.322171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.322021    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90657cd0-e12f-4ebe-90af-f63fee3e3105-console-oauth-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:29:15.888064 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.888029    2566 generic.go:356] "Generic (PLEG): container finished" podID="90657cd0-e12f-4ebe-90af-f63fee3e3105" containerID="516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5" exitCode=0
Sep 15 16:29:15.888494 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.888087    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55859b45fc-tjnwx" event={"ID":"90657cd0-e12f-4ebe-90af-f63fee3e3105","Type":"ContainerDied","Data":"516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5"}
Sep 15 16:29:15.888494 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.888091    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55859b45fc-tjnwx"
Sep 15 16:29:15.888494 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.888124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55859b45fc-tjnwx" event={"ID":"90657cd0-e12f-4ebe-90af-f63fee3e3105","Type":"ContainerDied","Data":"44273ea7c84b5c0cd480349a7bb14907958b068c225861ce93236b063ce0a374"}
Sep 15 16:29:15.888494 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.888143    2566 scope.go:117] "RemoveContainer" containerID="516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5"
Sep 15 16:29:15.896649 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.896632    2566 scope.go:117] "RemoveContainer" containerID="516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5"
Sep 15 16:29:15.896925 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:29:15.896906    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5\": container with ID starting with 516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5 not found: ID does not exist" containerID="516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5"
Sep 15 16:29:15.896963 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.896934    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5"} err="failed to get container status \"516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5\": rpc error: code = NotFound desc = could not find container \"516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5\": container with ID starting with 516d02c39a9aeff7d7b0a5ea6ec19b3fdfbc394a106f674f2e0075cb261450d5 not found: ID does not exist"
Sep 15 16:29:15.909611 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.909588    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55859b45fc-tjnwx"]
Sep 15 16:29:15.912906 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:15.912886    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55859b45fc-tjnwx"]
Sep 15 16:29:16.158433 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:16.158359    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90657cd0-e12f-4ebe-90af-f63fee3e3105" path="/var/lib/kubelet/pods/90657cd0-e12f-4ebe-90af-f63fee3e3105/volumes"
Sep 15 16:29:32.507751 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.507715    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b57f9b496-5scv8"]
Sep 15 16:29:32.508137 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.508001    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8df29ac7-55fb-42c0-bec2-f5375a8543ac" containerName="registry"
Sep 15 16:29:32.508137 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.508013    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8df29ac7-55fb-42c0-bec2-f5375a8543ac" containerName="registry"
Sep 15 16:29:32.508137 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.508028    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90657cd0-e12f-4ebe-90af-f63fee3e3105" containerName="console"
Sep 15 16:29:32.508137 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.508034    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90657cd0-e12f-4ebe-90af-f63fee3e3105" containerName="console"
Sep 15 16:29:32.508137 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.508090    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90657cd0-e12f-4ebe-90af-f63fee3e3105" containerName="console"
Sep 15 16:29:32.508137 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.508098    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8df29ac7-55fb-42c0-bec2-f5375a8543ac" containerName="registry"
Sep 15 16:29:32.512997 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.512968    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.521857 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.521827    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b57f9b496-5scv8"]
Sep 15 16:29:32.537608 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.537558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-service-ca\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.537782 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.537622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-trusted-ca-bundle\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.537782 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.537650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-684d6\" (UniqueName: \"kubernetes.io/projected/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-kube-api-access-684d6\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.537782 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.537725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-oauth-config\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.537782 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.537776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-config\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.537953 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.537804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-serving-cert\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.537953 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.537832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-oauth-serving-cert\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.639268 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.639233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-config\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.639268 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.639269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-serving-cert\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.639491 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.639291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-oauth-serving-cert\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.639491 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.639371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-service-ca\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.639491 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.639396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-trusted-ca-bundle\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.639491 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.639420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-684d6\" (UniqueName: \"kubernetes.io/projected/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-kube-api-access-684d6\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.639491 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.639465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-oauth-config\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.640229 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.640196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-service-ca\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.640532 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.640508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-oauth-serving-cert\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.640654 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.640508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-config\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.640654 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.640543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-trusted-ca-bundle\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.642394 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.642363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-serving-cert\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.642493 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.642471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-oauth-config\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.649047 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.649018    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-684d6\" (UniqueName: \"kubernetes.io/projected/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-kube-api-access-684d6\") pod \"console-5b57f9b496-5scv8\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") " pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:32.824046 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:32.823961    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:33.093795 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:33.093716    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b57f9b496-5scv8"]
Sep 15 16:29:33.096936 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:29:33.096906    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be1b87d_1653_4fbe_a8e8_7d3444aa8571.slice/crio-450e0139d4d2edd3905e75580b212a4b539b36f5a80ff984d9bf924e1dc456dc WatchSource:0}: Error finding container 450e0139d4d2edd3905e75580b212a4b539b36f5a80ff984d9bf924e1dc456dc: Status 404 returned error can't find the container with id 450e0139d4d2edd3905e75580b212a4b539b36f5a80ff984d9bf924e1dc456dc
Sep 15 16:29:33.959286 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:33.959252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b57f9b496-5scv8" event={"ID":"4be1b87d-1653-4fbe-a8e8-7d3444aa8571","Type":"ContainerStarted","Data":"a3e2c28782490d6985f9208f54ecf0c358a97fb74ed5efa439012aaf1176982d"}
Sep 15 16:29:33.959674 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:33.959301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b57f9b496-5scv8" event={"ID":"4be1b87d-1653-4fbe-a8e8-7d3444aa8571","Type":"ContainerStarted","Data":"450e0139d4d2edd3905e75580b212a4b539b36f5a80ff984d9bf924e1dc456dc"}
Sep 15 16:29:33.977301 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:33.977251    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b57f9b496-5scv8" podStartSLOduration=1.97723326 podStartE2EDuration="1.97723326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:29:33.976910269 +0000 UTC m=+162.344735494" watchObservedRunningTime="2026-09-15 16:29:33.97723326 +0000 UTC m=+162.345058459"
Sep 15 16:29:42.824881 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:42.824845    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:42.825317 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:42.825096    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:42.829393 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:42.829371    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:42.997494 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:42.997466    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:29:43.041275 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:29:43.041243    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647d99ddbf-n8525"]
Sep 15 16:30:08.064964 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.064900    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-647d99ddbf-n8525" podUID="c113c6c4-8457-4a77-825a-c879a6fe75b2" containerName="console" containerID="cri-o://ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485" gracePeriod=15
Sep 15 16:30:08.322899 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.322830    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:30:08.431532 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.431496    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-serving-cert\") pod \"c113c6c4-8457-4a77-825a-c879a6fe75b2\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") "
Sep 15 16:30:08.431717 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.431590    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-config\") pod \"c113c6c4-8457-4a77-825a-c879a6fe75b2\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") "
Sep 15 16:30:08.431763 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.431722    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-service-ca\") pod \"c113c6c4-8457-4a77-825a-c879a6fe75b2\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") "
Sep 15 16:30:08.431806 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.431776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-oauth-config\") pod \"c113c6c4-8457-4a77-825a-c879a6fe75b2\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") "
Sep 15 16:30:08.431848 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.431802    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-trusted-ca-bundle\") pod \"c113c6c4-8457-4a77-825a-c879a6fe75b2\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") "
Sep 15 16:30:08.431925 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.431903    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-oauth-serving-cert\") pod \"c113c6c4-8457-4a77-825a-c879a6fe75b2\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") "
Sep 15 16:30:08.431981 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.431941    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vk7x8\" (UniqueName: \"kubernetes.io/projected/c113c6c4-8457-4a77-825a-c879a6fe75b2-kube-api-access-vk7x8\") pod \"c113c6c4-8457-4a77-825a-c879a6fe75b2\" (UID: \"c113c6c4-8457-4a77-825a-c879a6fe75b2\") "
Sep 15 16:30:08.432034 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.432007    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-config" (OuterVolumeSpecName: "console-config") pod "c113c6c4-8457-4a77-825a-c879a6fe75b2" (UID: "c113c6c4-8457-4a77-825a-c879a6fe75b2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:30:08.432125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.432095    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-service-ca" (OuterVolumeSpecName: "service-ca") pod "c113c6c4-8457-4a77-825a-c879a6fe75b2" (UID: "c113c6c4-8457-4a77-825a-c879a6fe75b2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:30:08.432264 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.432192    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c113c6c4-8457-4a77-825a-c879a6fe75b2" (UID: "c113c6c4-8457-4a77-825a-c879a6fe75b2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:30:08.432264 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.432196    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:30:08.432387 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.432277    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c113c6c4-8457-4a77-825a-c879a6fe75b2" (UID: "c113c6c4-8457-4a77-825a-c879a6fe75b2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:30:08.433953 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.433928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c113c6c4-8457-4a77-825a-c879a6fe75b2" (UID: "c113c6c4-8457-4a77-825a-c879a6fe75b2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:30:08.433953 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.433943    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c113c6c4-8457-4a77-825a-c879a6fe75b2" (UID: "c113c6c4-8457-4a77-825a-c879a6fe75b2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:30:08.434066 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.433964    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c113c6c4-8457-4a77-825a-c879a6fe75b2-kube-api-access-vk7x8" (OuterVolumeSpecName: "kube-api-access-vk7x8") pod "c113c6c4-8457-4a77-825a-c879a6fe75b2" (UID: "c113c6c4-8457-4a77-825a-c879a6fe75b2"). InnerVolumeSpecName "kube-api-access-vk7x8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:30:08.533412 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.533364    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-serving-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:30:08.533412 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.533407    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-service-ca\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:30:08.533412 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.533418    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c113c6c4-8457-4a77-825a-c879a6fe75b2-console-oauth-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:30:08.533412 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.533426    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-trusted-ca-bundle\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:30:08.533704 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.533435    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c113c6c4-8457-4a77-825a-c879a6fe75b2-oauth-serving-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:30:08.533704 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:08.533443    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk7x8\" (UniqueName: \"kubernetes.io/projected/c113c6c4-8457-4a77-825a-c879a6fe75b2-kube-api-access-vk7x8\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:30:09.088172 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.088134    2566 generic.go:356] "Generic (PLEG): container finished" podID="c113c6c4-8457-4a77-825a-c879a6fe75b2" containerID="ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485" exitCode=0
Sep 15 16:30:09.088607 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.088189    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-647d99ddbf-n8525"
Sep 15 16:30:09.088607 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.088218    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647d99ddbf-n8525" event={"ID":"c113c6c4-8457-4a77-825a-c879a6fe75b2","Type":"ContainerDied","Data":"ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485"}
Sep 15 16:30:09.088607 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.088269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-647d99ddbf-n8525" event={"ID":"c113c6c4-8457-4a77-825a-c879a6fe75b2","Type":"ContainerDied","Data":"e8c942d220d480c8db452f4ce7fddb2c4a5bc5181d2ff828013bec7c655f90fc"}
Sep 15 16:30:09.088607 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.088292    2566 scope.go:117] "RemoveContainer" containerID="ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485"
Sep 15 16:30:09.096587 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.096550    2566 scope.go:117] "RemoveContainer" containerID="ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485"
Sep 15 16:30:09.096863 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:30:09.096844    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485\": container with ID starting with ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485 not found: ID does not exist" containerID="ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485"
Sep 15 16:30:09.096934 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.096871    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485"} err="failed to get container status \"ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485\": rpc error: code = NotFound desc = could not find container \"ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485\": container with ID starting with ac9f68038cbd44270c52df11728dfd0c09679c9a50db79982b0cded5dd0e9485 not found: ID does not exist"
Sep 15 16:30:09.106785 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.106755    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-647d99ddbf-n8525"]
Sep 15 16:30:09.107928 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:09.107910    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-647d99ddbf-n8525"]
Sep 15 16:30:10.157831 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:10.157797    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c113c6c4-8457-4a77-825a-c879a6fe75b2" path="/var/lib/kubelet/pods/c113c6c4-8457-4a77-825a-c879a6fe75b2/volumes"
Sep 15 16:30:56.569442 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.569398    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"]
Sep 15 16:30:56.569974 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.569835    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c113c6c4-8457-4a77-825a-c879a6fe75b2" containerName="console"
Sep 15 16:30:56.569974 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.569854    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c113c6c4-8457-4a77-825a-c879a6fe75b2" containerName="console"
Sep 15 16:30:56.569974 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.569948    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c113c6c4-8457-4a77-825a-c879a6fe75b2" containerName="console"
Sep 15 16:30:56.574194 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.574173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.576774 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.576743    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-xbf6b\""
Sep 15 16:30:56.576774 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.576772    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 15 16:30:56.576914 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.576826    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 15 16:30:56.582412 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.582388    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"]
Sep 15 16:30:56.694854 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.694813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.695036 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.694867    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4q45\" (UniqueName: \"kubernetes.io/projected/9c276c8d-36f6-47d5-8a74-123fd9014134-kube-api-access-r4q45\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.695036 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.694929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.796300 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.796256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.796300 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.796306    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q45\" (UniqueName: \"kubernetes.io/projected/9c276c8d-36f6-47d5-8a74-123fd9014134-kube-api-access-r4q45\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.796514 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.796333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.796721 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.796702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.796768 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.796740    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.806930 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.806907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4q45\" (UniqueName: \"kubernetes.io/projected/9c276c8d-36f6-47d5-8a74-123fd9014134-kube-api-access-r4q45\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:56.883945 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:56.883914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:30:57.153256 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:57.153232    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"]
Sep 15 16:30:57.155742 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:30:57.155708    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c276c8d_36f6_47d5_8a74_123fd9014134.slice/crio-512bfe668aa79b1865c2b7ee891c3ff046aad276abbfbef98f1b67d1e2763235 WatchSource:0}: Error finding container 512bfe668aa79b1865c2b7ee891c3ff046aad276abbfbef98f1b67d1e2763235: Status 404 returned error can't find the container with id 512bfe668aa79b1865c2b7ee891c3ff046aad276abbfbef98f1b67d1e2763235
Sep 15 16:30:57.270848 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:30:57.270812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp" event={"ID":"9c276c8d-36f6-47d5-8a74-123fd9014134","Type":"ContainerStarted","Data":"512bfe668aa79b1865c2b7ee891c3ff046aad276abbfbef98f1b67d1e2763235"}
Sep 15 16:31:03.297400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:03.297362    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerID="7809ac63c267510e859fcf8094c695ccaca6693a86142c449a09984ec5fb0be3" exitCode=0
Sep 15 16:31:03.297827 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:03.297425    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp" event={"ID":"9c276c8d-36f6-47d5-8a74-123fd9014134","Type":"ContainerDied","Data":"7809ac63c267510e859fcf8094c695ccaca6693a86142c449a09984ec5fb0be3"}
Sep 15 16:31:06.311065 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:06.311025    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerID="ac67bdd991c4e365ca13ff13415f640cf871c7097c712b658c3c931118ea6bd6" exitCode=0
Sep 15 16:31:06.311449 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:06.311098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp" event={"ID":"9c276c8d-36f6-47d5-8a74-123fd9014134","Type":"ContainerDied","Data":"ac67bdd991c4e365ca13ff13415f640cf871c7097c712b658c3c931118ea6bd6"}
Sep 15 16:31:13.341966 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:13.341929    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerID="bf010e3540b253dbc1f4fc84b13aae4ea75d38eb173e532c4b1f0314efd8767e" exitCode=0
Sep 15 16:31:13.342316 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:13.341989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp" event={"ID":"9c276c8d-36f6-47d5-8a74-123fd9014134","Type":"ContainerDied","Data":"bf010e3540b253dbc1f4fc84b13aae4ea75d38eb173e532c4b1f0314efd8767e"}
Sep 15 16:31:14.466210 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.466186    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:31:14.542695 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.542655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4q45\" (UniqueName: \"kubernetes.io/projected/9c276c8d-36f6-47d5-8a74-123fd9014134-kube-api-access-r4q45\") pod \"9c276c8d-36f6-47d5-8a74-123fd9014134\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") "
Sep 15 16:31:14.542890 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.542718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-bundle\") pod \"9c276c8d-36f6-47d5-8a74-123fd9014134\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") "
Sep 15 16:31:14.542890 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.542781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-util\") pod \"9c276c8d-36f6-47d5-8a74-123fd9014134\" (UID: \"9c276c8d-36f6-47d5-8a74-123fd9014134\") "
Sep 15 16:31:14.543533 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.543501    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-bundle" (OuterVolumeSpecName: "bundle") pod "9c276c8d-36f6-47d5-8a74-123fd9014134" (UID: "9c276c8d-36f6-47d5-8a74-123fd9014134"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:31:14.545018 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.544986    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c276c8d-36f6-47d5-8a74-123fd9014134-kube-api-access-r4q45" (OuterVolumeSpecName: "kube-api-access-r4q45") pod "9c276c8d-36f6-47d5-8a74-123fd9014134" (UID: "9c276c8d-36f6-47d5-8a74-123fd9014134"). InnerVolumeSpecName "kube-api-access-r4q45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:31:14.547381 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.547356    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-util" (OuterVolumeSpecName: "util") pod "9c276c8d-36f6-47d5-8a74-123fd9014134" (UID: "9c276c8d-36f6-47d5-8a74-123fd9014134"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:31:14.643327 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.643292    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-util\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:31:14.643327 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.643324    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4q45\" (UniqueName: \"kubernetes.io/projected/9c276c8d-36f6-47d5-8a74-123fd9014134-kube-api-access-r4q45\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:31:14.643515 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:14.643335    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9c276c8d-36f6-47d5-8a74-123fd9014134-bundle\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:31:15.352364 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:15.352332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp" event={"ID":"9c276c8d-36f6-47d5-8a74-123fd9014134","Type":"ContainerDied","Data":"512bfe668aa79b1865c2b7ee891c3ff046aad276abbfbef98f1b67d1e2763235"}
Sep 15 16:31:15.352524 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:15.352379    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="512bfe668aa79b1865c2b7ee891c3ff046aad276abbfbef98f1b67d1e2763235"
Sep 15 16:31:15.352524 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:15.352388    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2d5dp"
Sep 15 16:31:18.841860 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.841821    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"]
Sep 15 16:31:18.842233 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.842096    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerName="pull"
Sep 15 16:31:18.842233 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.842106    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerName="pull"
Sep 15 16:31:18.842233 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.842121    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerName="util"
Sep 15 16:31:18.842233 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.842126    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerName="util"
Sep 15 16:31:18.842233 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.842135    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerName="extract"
Sep 15 16:31:18.842233 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.842141    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerName="extract"
Sep 15 16:31:18.842233 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.842189    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c276c8d-36f6-47d5-8a74-123fd9014134" containerName="extract"
Sep 15 16:31:18.844116 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.844098    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:18.847483 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.847457    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 15 16:31:18.847483 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.847481    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-wwf7w\""
Sep 15 16:31:18.847673 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.847486    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 15 16:31:18.847673 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.847601    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 15 16:31:18.856214 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.856186    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"]
Sep 15 16:31:18.977013 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.976971    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvkcn\" (UniqueName: \"kubernetes.io/projected/3849df61-c186-426e-8cc6-e8d5b26b78fe-kube-api-access-fvkcn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-x5k5k\" (UID: \"3849df61-c186-426e-8cc6-e8d5b26b78fe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:18.977189 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:18.977029    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3849df61-c186-426e-8cc6-e8d5b26b78fe-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-x5k5k\" (UID: \"3849df61-c186-426e-8cc6-e8d5b26b78fe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:19.078398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:19.078349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvkcn\" (UniqueName: \"kubernetes.io/projected/3849df61-c186-426e-8cc6-e8d5b26b78fe-kube-api-access-fvkcn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-x5k5k\" (UID: \"3849df61-c186-426e-8cc6-e8d5b26b78fe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:19.078540 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:19.078416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3849df61-c186-426e-8cc6-e8d5b26b78fe-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-x5k5k\" (UID: \"3849df61-c186-426e-8cc6-e8d5b26b78fe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:19.080895 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:19.080858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/3849df61-c186-426e-8cc6-e8d5b26b78fe-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-x5k5k\" (UID: \"3849df61-c186-426e-8cc6-e8d5b26b78fe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:19.093086 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:19.093017    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvkcn\" (UniqueName: \"kubernetes.io/projected/3849df61-c186-426e-8cc6-e8d5b26b78fe-kube-api-access-fvkcn\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-x5k5k\" (UID: \"3849df61-c186-426e-8cc6-e8d5b26b78fe\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:19.154075 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:19.154035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:19.425619 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:19.425592    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"]
Sep 15 16:31:19.426940 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:31:19.426918    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3849df61_c186_426e_8cc6_e8d5b26b78fe.slice/crio-05dd6aaa2d7e558824ee0f085b221f31c8619bda0f69af8321806fa0bd169955 WatchSource:0}: Error finding container 05dd6aaa2d7e558824ee0f085b221f31c8619bda0f69af8321806fa0bd169955: Status 404 returned error can't find the container with id 05dd6aaa2d7e558824ee0f085b221f31c8619bda0f69af8321806fa0bd169955
Sep 15 16:31:20.373788 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:20.373738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k" event={"ID":"3849df61-c186-426e-8cc6-e8d5b26b78fe","Type":"ContainerStarted","Data":"05dd6aaa2d7e558824ee0f085b221f31c8619bda0f69af8321806fa0bd169955"}
Sep 15 16:31:22.384050 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:22.384007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k" event={"ID":"3849df61-c186-426e-8cc6-e8d5b26b78fe","Type":"ContainerStarted","Data":"0e231a1a8617ed25815e402c812ade9aded9473ea53078598a6cf57af5e5e112"}
Sep 15 16:31:22.384431 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:22.384266    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:22.407207 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:22.407149    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k" podStartSLOduration=1.546396551 podStartE2EDuration="4.407131815s" totalImagesPullingTime="2.860735264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:31:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:31:22.405102618 +0000 UTC m=+270.772927840" watchObservedRunningTime="2026-09-15 16:31:22.407131815 +0000 UTC m=+270.774957014"
Sep 15 16:31:23.063643 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.063604    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-tgxq4"]
Sep 15 16:31:23.065853 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.065829    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.068230 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.068208    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 15 16:31:23.068351 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.068212    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 15 16:31:23.068351 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.068214    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-5nnpl\""
Sep 15 16:31:23.077164 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.077139    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-tgxq4"]
Sep 15 16:31:23.213268 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.213236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-certificates\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.213268 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.213268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-cabundle0\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.213482 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.213290    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqmq\" (UniqueName: \"kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-kube-api-access-jcqmq\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.314815 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.314726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-certificates\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.314815 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.314763    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-cabundle0\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.314815 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.314792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqmq\" (UniqueName: \"kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-kube-api-access-jcqmq\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.315040 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:31:23.314882    2566 secret.go:281] references non-existent secret key: ca.crt
Sep 15 16:31:23.315040 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:31:23.314905    2566 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 15 16:31:23.315040 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:31:23.314919    2566 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-tgxq4: references non-existent secret key: ca.crt
Sep 15 16:31:23.315040 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:31:23.314976    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-certificates podName:96d8ebc4-6976-4e81-8da7-cb2fbf4cc996 nodeName:}" failed. No retries permitted until 2026-09-15 16:31:23.814960113 +0000 UTC m=+272.182785290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-certificates") pod "keda-operator-579c8958f7-tgxq4" (UID: "96d8ebc4-6976-4e81-8da7-cb2fbf4cc996") : references non-existent secret key: ca.crt
Sep 15 16:31:23.315434 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.315417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-cabundle0\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.325186 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.325162    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqmq\" (UniqueName: \"kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-kube-api-access-jcqmq\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.413107 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.413078    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"]
Sep 15 16:31:23.415503 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.415481    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.417579 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.417547    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 15 16:31:23.425052 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.425029    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"]
Sep 15 16:31:23.517090 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.517043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b558402b-e16d-4423-88ea-e142ec900c86-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.517280 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.517172    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sc5f\" (UniqueName: \"kubernetes.io/projected/b558402b-e16d-4423-88ea-e142ec900c86-kube-api-access-5sc5f\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.517280 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.517242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b558402b-e16d-4423-88ea-e142ec900c86-certificates\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.617188 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.617114    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-r28qp"]
Sep 15 16:31:23.617940 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.617889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b558402b-e16d-4423-88ea-e142ec900c86-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.618045 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.617985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sc5f\" (UniqueName: \"kubernetes.io/projected/b558402b-e16d-4423-88ea-e142ec900c86-kube-api-access-5sc5f\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.618045 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.618021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b558402b-e16d-4423-88ea-e142ec900c86-certificates\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.618308 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.618286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/b558402b-e16d-4423-88ea-e142ec900c86-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.619719 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.619698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:23.620595 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.620541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/b558402b-e16d-4423-88ea-e142ec900c86-certificates\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.621969 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.621955    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 15 16:31:23.625479 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.625459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sc5f\" (UniqueName: \"kubernetes.io/projected/b558402b-e16d-4423-88ea-e142ec900c86-kube-api-access-5sc5f\") pod \"keda-metrics-apiserver-77857b7b98-b9dcp\" (UID: \"b558402b-e16d-4423-88ea-e142ec900c86\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.627920 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.627898    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-r28qp"]
Sep 15 16:31:23.718960 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.718922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnmtn\" (UniqueName: \"kubernetes.io/projected/a09b658e-0e36-41ce-a569-2d8a633112d0-kube-api-access-nnmtn\") pod \"keda-admission-6c4d864b86-r28qp\" (UID: \"a09b658e-0e36-41ce-a569-2d8a633112d0\") " pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:23.719115 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.718968    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a09b658e-0e36-41ce-a569-2d8a633112d0-certificates\") pod \"keda-admission-6c4d864b86-r28qp\" (UID: \"a09b658e-0e36-41ce-a569-2d8a633112d0\") " pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:23.726906 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.726868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:23.819887 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.819849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-certificates\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.820034 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.819963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnmtn\" (UniqueName: \"kubernetes.io/projected/a09b658e-0e36-41ce-a569-2d8a633112d0-kube-api-access-nnmtn\") pod \"keda-admission-6c4d864b86-r28qp\" (UID: \"a09b658e-0e36-41ce-a569-2d8a633112d0\") " pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:23.820034 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.820009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a09b658e-0e36-41ce-a569-2d8a633112d0-certificates\") pod \"keda-admission-6c4d864b86-r28qp\" (UID: \"a09b658e-0e36-41ce-a569-2d8a633112d0\") " pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:23.822531 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.822493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/96d8ebc4-6976-4e81-8da7-cb2fbf4cc996-certificates\") pod \"keda-operator-579c8958f7-tgxq4\" (UID: \"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996\") " pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:23.822651 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.822493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a09b658e-0e36-41ce-a569-2d8a633112d0-certificates\") pod \"keda-admission-6c4d864b86-r28qp\" (UID: \"a09b658e-0e36-41ce-a569-2d8a633112d0\") " pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:23.827547 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.827526    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnmtn\" (UniqueName: \"kubernetes.io/projected/a09b658e-0e36-41ce-a569-2d8a633112d0-kube-api-access-nnmtn\") pod \"keda-admission-6c4d864b86-r28qp\" (UID: \"a09b658e-0e36-41ce-a569-2d8a633112d0\") " pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:23.944515 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.944478    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:23.976024 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:23.975981    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:24.001716 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:24.001528    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"]
Sep 15 16:31:24.006037 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:31:24.006006    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb558402b_e16d_4423_88ea_e142ec900c86.slice/crio-d5af4f99af8b8a71f26dbc2125b9c33de8c3867fbe61762de51327026d4e496d WatchSource:0}: Error finding container d5af4f99af8b8a71f26dbc2125b9c33de8c3867fbe61762de51327026d4e496d: Status 404 returned error can't find the container with id d5af4f99af8b8a71f26dbc2125b9c33de8c3867fbe61762de51327026d4e496d
Sep 15 16:31:24.220014 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:24.219984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-r28qp"]
Sep 15 16:31:24.222535 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:31:24.222502    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda09b658e_0e36_41ce_a569_2d8a633112d0.slice/crio-105c90705026a64f9ac7379eedb7136b63be76eeb78def7238d2061d031ded29 WatchSource:0}: Error finding container 105c90705026a64f9ac7379eedb7136b63be76eeb78def7238d2061d031ded29: Status 404 returned error can't find the container with id 105c90705026a64f9ac7379eedb7136b63be76eeb78def7238d2061d031ded29
Sep 15 16:31:24.255006 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:24.254973    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-tgxq4"]
Sep 15 16:31:24.257843 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:31:24.257815    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d8ebc4_6976_4e81_8da7_cb2fbf4cc996.slice/crio-bfe4a2512fa83d7aef0e516a407dba1687981be039ae8543a674580db333da51 WatchSource:0}: Error finding container bfe4a2512fa83d7aef0e516a407dba1687981be039ae8543a674580db333da51: Status 404 returned error can't find the container with id bfe4a2512fa83d7aef0e516a407dba1687981be039ae8543a674580db333da51
Sep 15 16:31:24.393273 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:24.393231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp" event={"ID":"b558402b-e16d-4423-88ea-e142ec900c86","Type":"ContainerStarted","Data":"d5af4f99af8b8a71f26dbc2125b9c33de8c3867fbe61762de51327026d4e496d"}
Sep 15 16:31:24.394317 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:24.394281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-r28qp" event={"ID":"a09b658e-0e36-41ce-a569-2d8a633112d0","Type":"ContainerStarted","Data":"105c90705026a64f9ac7379eedb7136b63be76eeb78def7238d2061d031ded29"}
Sep 15 16:31:24.395415 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:24.395395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-tgxq4" event={"ID":"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996","Type":"ContainerStarted","Data":"bfe4a2512fa83d7aef0e516a407dba1687981be039ae8543a674580db333da51"}
Sep 15 16:31:27.411151 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:27.411109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-r28qp" event={"ID":"a09b658e-0e36-41ce-a569-2d8a633112d0","Type":"ContainerStarted","Data":"068c9ba2699dff1b1608e380e53d1f242213414a347b3373cf0ed4aa3e200964"}
Sep 15 16:31:27.411600 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:27.411299    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:27.428934 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:27.428885    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-r28qp" podStartSLOduration=2.221072021 podStartE2EDuration="4.428869963s" totalImagesPullingTime="2.207797942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:31:27.427891531 +0000 UTC m=+275.795716729" watchObservedRunningTime="2026-09-15 16:31:27.428869963 +0000 UTC m=+275.796695539"
Sep 15 16:31:29.423047 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:29.423014    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-tgxq4" event={"ID":"96d8ebc4-6976-4e81-8da7-cb2fbf4cc996","Type":"ContainerStarted","Data":"f3cdbb66bd221cc7a76aa7e43958595f3c50d0afe81c5047f4464ad22fa22cca"}
Sep 15 16:31:29.423488 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:29.423146    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:29.424771 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:29.424733    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp" event={"ID":"b558402b-e16d-4423-88ea-e142ec900c86","Type":"ContainerStarted","Data":"ae581267e6fbdc52ed29675ba035bf329d056271ced6e7a6ec92ae9ec5f58c2e"}
Sep 15 16:31:29.424896 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:29.424877    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:29.439740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:29.439697    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-tgxq4" podStartSLOduration=1.827781267 podStartE2EDuration="6.439684778s" totalImagesPullingTime="4.611903511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:31:29.43840036 +0000 UTC m=+277.806225561" watchObservedRunningTime="2026-09-15 16:31:29.439684778 +0000 UTC m=+277.807509977"
Sep 15 16:31:29.453907 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:29.453858    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp" podStartSLOduration=1.594368314 podStartE2EDuration="6.453841368s" totalImagesPullingTime="4.859473054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:31:29.453243199 +0000 UTC m=+277.821068401" watchObservedRunningTime="2026-09-15 16:31:29.453841368 +0000 UTC m=+277.821666567"
Sep 15 16:31:40.437041 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:40.437006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-b9dcp"
Sep 15 16:31:43.391135 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:43.391096    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-x5k5k"
Sep 15 16:31:48.419286 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:48.419255    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-r28qp"
Sep 15 16:31:50.434730 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:50.434696    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-tgxq4"
Sep 15 16:31:52.031492 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:52.031461    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:31:52.032976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:52.032949    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:31:52.042226 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:31:52.042204    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 15 16:32:43.376064 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.376028    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-97hl8"]
Sep 15 16:32:43.378323 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.378305    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:43.380932 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.380909    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 15 16:32:43.381071 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.380913    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 15 16:32:43.381687 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.381670    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 15 16:32:43.381772 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.381722    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-h5tz4\""
Sep 15 16:32:43.393585 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.393552    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-97hl8"]
Sep 15 16:32:43.417419 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.417390    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-2mkmz"]
Sep 15 16:32:43.419553 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.419531    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:43.422076 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.422056    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-d4bf6\""
Sep 15 16:32:43.422190 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.422087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 15 16:32:43.429002 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.428977    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2mkmz"]
Sep 15 16:32:43.477923 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.477885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-cert\") pod \"kserve-controller-manager-b589c8cc8-97hl8\" (UID: \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\") " pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:43.478098 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.477981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrpvb\" (UniqueName: \"kubernetes.io/projected/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-kube-api-access-xrpvb\") pod \"kserve-controller-manager-b589c8cc8-97hl8\" (UID: \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\") " pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:43.578954 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.578913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-cert\") pod \"kserve-controller-manager-b589c8cc8-97hl8\" (UID: \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\") " pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:43.579138 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.578964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8tm\" (UniqueName: \"kubernetes.io/projected/353b6b59-1a28-4129-b4ca-4d4f5056b713-kube-api-access-vx8tm\") pod \"seaweedfs-64568bcd49-2mkmz\" (UID: \"353b6b59-1a28-4129-b4ca-4d4f5056b713\") " pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:43.579138 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.579017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/353b6b59-1a28-4129-b4ca-4d4f5056b713-data\") pod \"seaweedfs-64568bcd49-2mkmz\" (UID: \"353b6b59-1a28-4129-b4ca-4d4f5056b713\") " pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:43.579138 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.579046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrpvb\" (UniqueName: \"kubernetes.io/projected/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-kube-api-access-xrpvb\") pod \"kserve-controller-manager-b589c8cc8-97hl8\" (UID: \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\") " pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:43.581379 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.581344    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-cert\") pod \"kserve-controller-manager-b589c8cc8-97hl8\" (UID: \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\") " pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:43.587722 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.587699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrpvb\" (UniqueName: \"kubernetes.io/projected/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-kube-api-access-xrpvb\") pod \"kserve-controller-manager-b589c8cc8-97hl8\" (UID: \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\") " pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:43.679524 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.679489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8tm\" (UniqueName: \"kubernetes.io/projected/353b6b59-1a28-4129-b4ca-4d4f5056b713-kube-api-access-vx8tm\") pod \"seaweedfs-64568bcd49-2mkmz\" (UID: \"353b6b59-1a28-4129-b4ca-4d4f5056b713\") " pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:43.679710 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.679565    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/353b6b59-1a28-4129-b4ca-4d4f5056b713-data\") pod \"seaweedfs-64568bcd49-2mkmz\" (UID: \"353b6b59-1a28-4129-b4ca-4d4f5056b713\") " pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:43.679940 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.679922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/353b6b59-1a28-4129-b4ca-4d4f5056b713-data\") pod \"seaweedfs-64568bcd49-2mkmz\" (UID: \"353b6b59-1a28-4129-b4ca-4d4f5056b713\") " pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:43.688272 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.688249    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:43.688754 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.688734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8tm\" (UniqueName: \"kubernetes.io/projected/353b6b59-1a28-4129-b4ca-4d4f5056b713-kube-api-access-vx8tm\") pod \"seaweedfs-64568bcd49-2mkmz\" (UID: \"353b6b59-1a28-4129-b4ca-4d4f5056b713\") " pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:43.728232 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.728202    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:43.959936 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.959817    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-97hl8"]
Sep 15 16:32:43.962610 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:32:43.962581    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd03dd4b_34cf_4593_a84d_78c7b2f9d351.slice/crio-9f501980a5006f5ccf31956ff9e5c281ac00d029b2ab70aa92d4306efb6c4e55 WatchSource:0}: Error finding container 9f501980a5006f5ccf31956ff9e5c281ac00d029b2ab70aa92d4306efb6c4e55: Status 404 returned error can't find the container with id 9f501980a5006f5ccf31956ff9e5c281ac00d029b2ab70aa92d4306efb6c4e55
Sep 15 16:32:43.963942 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:43.963926    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:32:44.005546 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:44.005520    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-2mkmz"]
Sep 15 16:32:44.007987 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:32:44.007964    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod353b6b59_1a28_4129_b4ca_4d4f5056b713.slice/crio-56adefe2cd2bb79cd3b8808b446389164f8f89c045694c9d13d72b78e51fda08 WatchSource:0}: Error finding container 56adefe2cd2bb79cd3b8808b446389164f8f89c045694c9d13d72b78e51fda08: Status 404 returned error can't find the container with id 56adefe2cd2bb79cd3b8808b446389164f8f89c045694c9d13d72b78e51fda08
Sep 15 16:32:44.750459 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:44.750421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8" event={"ID":"dd03dd4b-34cf-4593-a84d-78c7b2f9d351","Type":"ContainerStarted","Data":"9f501980a5006f5ccf31956ff9e5c281ac00d029b2ab70aa92d4306efb6c4e55"}
Sep 15 16:32:44.752259 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:44.752201    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2mkmz" event={"ID":"353b6b59-1a28-4129-b4ca-4d4f5056b713","Type":"ContainerStarted","Data":"56adefe2cd2bb79cd3b8808b446389164f8f89c045694c9d13d72b78e51fda08"}
Sep 15 16:32:47.769150 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:47.769108    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8" event={"ID":"dd03dd4b-34cf-4593-a84d-78c7b2f9d351","Type":"ContainerStarted","Data":"c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62"}
Sep 15 16:32:47.769639 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:47.769336    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:32:47.786283 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:47.786203    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8" podStartSLOduration=2.039221147 podStartE2EDuration="4.786189871s" totalImagesPullingTime="2.746968724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:32:47.785475117 +0000 UTC m=+356.153300317" watchObservedRunningTime="2026-09-15 16:32:47.786189871 +0000 UTC m=+356.154015070"
Sep 15 16:32:48.775052 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:48.775015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-2mkmz" event={"ID":"353b6b59-1a28-4129-b4ca-4d4f5056b713","Type":"ContainerStarted","Data":"3e1d9d8a628d50e0b3214664eff495ba53a72ce417238fe3ce1a3865bc0e387b"}
Sep 15 16:32:48.775409 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:48.775266    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:32:48.792696 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:48.792653    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-2mkmz" podStartSLOduration=1.186258188 podStartE2EDuration="5.792639943s" totalImagesPullingTime="4.606381755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:32:48.791126244 +0000 UTC m=+357.158951443" watchObservedRunningTime="2026-09-15 16:32:48.792639943 +0000 UTC m=+357.160465153"
Sep 15 16:32:54.781497 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:32:54.781461    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-2mkmz"
Sep 15 16:33:18.781277 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:18.781241    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:33:27.028645 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.028611    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-97hl8"]
Sep 15 16:33:27.029059 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.028809    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8" podUID="dd03dd4b-34cf-4593-a84d-78c7b2f9d351" containerName="manager" containerID="cri-o://c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62" gracePeriod=10
Sep 15 16:33:27.053611 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.053583    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-lzm92"]
Sep 15 16:33:27.055865 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.055849    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:27.068124 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.068099    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-lzm92"]
Sep 15 16:33:27.131749 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.131719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d44f95fb-f12a-415d-bd5d-affde1179526-cert\") pod \"kserve-controller-manager-b589c8cc8-lzm92\" (UID: \"d44f95fb-f12a-415d-bd5d-affde1179526\") " pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:27.131903 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.131804    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rcjv\" (UniqueName: \"kubernetes.io/projected/d44f95fb-f12a-415d-bd5d-affde1179526-kube-api-access-9rcjv\") pod \"kserve-controller-manager-b589c8cc8-lzm92\" (UID: \"d44f95fb-f12a-415d-bd5d-affde1179526\") " pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:27.233136 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.233098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rcjv\" (UniqueName: \"kubernetes.io/projected/d44f95fb-f12a-415d-bd5d-affde1179526-kube-api-access-9rcjv\") pod \"kserve-controller-manager-b589c8cc8-lzm92\" (UID: \"d44f95fb-f12a-415d-bd5d-affde1179526\") " pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:27.233319 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.233213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d44f95fb-f12a-415d-bd5d-affde1179526-cert\") pod \"kserve-controller-manager-b589c8cc8-lzm92\" (UID: \"d44f95fb-f12a-415d-bd5d-affde1179526\") " pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:27.235546 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.235521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d44f95fb-f12a-415d-bd5d-affde1179526-cert\") pod \"kserve-controller-manager-b589c8cc8-lzm92\" (UID: \"d44f95fb-f12a-415d-bd5d-affde1179526\") " pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:27.242051 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.242023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rcjv\" (UniqueName: \"kubernetes.io/projected/d44f95fb-f12a-415d-bd5d-affde1179526-kube-api-access-9rcjv\") pod \"kserve-controller-manager-b589c8cc8-lzm92\" (UID: \"d44f95fb-f12a-415d-bd5d-affde1179526\") " pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:27.267402 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.267380    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:33:27.333835 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.333745    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-cert\") pod \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\" (UID: \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\") "
Sep 15 16:33:27.333835 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.333827    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrpvb\" (UniqueName: \"kubernetes.io/projected/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-kube-api-access-xrpvb\") pod \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\" (UID: \"dd03dd4b-34cf-4593-a84d-78c7b2f9d351\") "
Sep 15 16:33:27.336059 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.336023    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-cert" (OuterVolumeSpecName: "cert") pod "dd03dd4b-34cf-4593-a84d-78c7b2f9d351" (UID: "dd03dd4b-34cf-4593-a84d-78c7b2f9d351"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:33:27.336194 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.336060    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-kube-api-access-xrpvb" (OuterVolumeSpecName: "kube-api-access-xrpvb") pod "dd03dd4b-34cf-4593-a84d-78c7b2f9d351" (UID: "dd03dd4b-34cf-4593-a84d-78c7b2f9d351"). InnerVolumeSpecName "kube-api-access-xrpvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:33:27.411240 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.411199    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:27.434470 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.434441    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrpvb\" (UniqueName: \"kubernetes.io/projected/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-kube-api-access-xrpvb\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:33:27.434470 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.434470    2566 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd03dd4b-34cf-4593-a84d-78c7b2f9d351-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:33:27.679226 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.679192    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-lzm92"]
Sep 15 16:33:27.683506 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:33:27.683483    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44f95fb_f12a_415d_bd5d_affde1179526.slice/crio-3c1ee1cd7d2e061f032ab632a350679f2a3b9d083625a80ede28a7f21c4edc47 WatchSource:0}: Error finding container 3c1ee1cd7d2e061f032ab632a350679f2a3b9d083625a80ede28a7f21c4edc47: Status 404 returned error can't find the container with id 3c1ee1cd7d2e061f032ab632a350679f2a3b9d083625a80ede28a7f21c4edc47
Sep 15 16:33:27.942920 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.942829    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd03dd4b-34cf-4593-a84d-78c7b2f9d351" containerID="c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62" exitCode=0
Sep 15 16:33:27.942920 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.942881    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8"
Sep 15 16:33:27.942920 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.942908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8" event={"ID":"dd03dd4b-34cf-4593-a84d-78c7b2f9d351","Type":"ContainerDied","Data":"c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62"}
Sep 15 16:33:27.943205 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.942972    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-97hl8" event={"ID":"dd03dd4b-34cf-4593-a84d-78c7b2f9d351","Type":"ContainerDied","Data":"9f501980a5006f5ccf31956ff9e5c281ac00d029b2ab70aa92d4306efb6c4e55"}
Sep 15 16:33:27.943205 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.943005    2566 scope.go:117] "RemoveContainer" containerID="c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62"
Sep 15 16:33:27.944363 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.944333    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-lzm92" event={"ID":"d44f95fb-f12a-415d-bd5d-affde1179526","Type":"ContainerStarted","Data":"3c1ee1cd7d2e061f032ab632a350679f2a3b9d083625a80ede28a7f21c4edc47"}
Sep 15 16:33:27.951663 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.951646    2566 scope.go:117] "RemoveContainer" containerID="c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62"
Sep 15 16:33:27.951930 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:33:27.951914    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62\": container with ID starting with c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62 not found: ID does not exist" containerID="c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62"
Sep 15 16:33:27.951975 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.951936    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62"} err="failed to get container status \"c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62\": rpc error: code = NotFound desc = could not find container \"c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62\": container with ID starting with c220304d7d5ac1dbc8ab1abade4a7e35f04fbff2b7027e45f35e6a8a42b67e62 not found: ID does not exist"
Sep 15 16:33:27.968804 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.968773    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-97hl8"]
Sep 15 16:33:27.976505 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:27.976484    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-b589c8cc8-97hl8"]
Sep 15 16:33:28.158971 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.158934    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd03dd4b-34cf-4593-a84d-78c7b2f9d351" path="/var/lib/kubelet/pods/dd03dd4b-34cf-4593-a84d-78c7b2f9d351/volumes"
Sep 15 16:33:28.734507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.734426    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-657d8c56dc-d5p7j"]
Sep 15 16:33:28.734816 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.734802    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd03dd4b-34cf-4593-a84d-78c7b2f9d351" containerName="manager"
Sep 15 16:33:28.734874 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.734817    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd03dd4b-34cf-4593-a84d-78c7b2f9d351" containerName="manager"
Sep 15 16:33:28.734911 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.734877    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd03dd4b-34cf-4593-a84d-78c7b2f9d351" containerName="manager"
Sep 15 16:33:28.736728 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.736708    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.749215 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.749190    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-657d8c56dc-d5p7j"]
Sep 15 16:33:28.846678 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.846634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-trusted-ca-bundle\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.846678 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.846685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-oauth-serving-cert\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.846898 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.846707    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-config\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.846898 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.846767    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-service-ca\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.846898 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.846786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-serving-cert\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.846898 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.846855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-oauth-config\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.846898 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.846894    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcq7t\" (UniqueName: \"kubernetes.io/projected/117d5a5e-d4a8-4274-9227-e6a687222bd6-kube-api-access-hcq7t\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.948052 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.948014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-service-ca\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.948244 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.948063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-serving-cert\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.948244 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.948167    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-oauth-config\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.948244 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.948204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcq7t\" (UniqueName: \"kubernetes.io/projected/117d5a5e-d4a8-4274-9227-e6a687222bd6-kube-api-access-hcq7t\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.948407 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.948253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-trusted-ca-bundle\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.948407 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.948273    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-oauth-serving-cert\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.948407 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.948303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-config\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.948876 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.948819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-service-ca\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.949037 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.949011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-oauth-serving-cert\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.949162 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.949136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-trusted-ca-bundle\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.949277 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.949263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-config\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.950817 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.950794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-oauth-config\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.950943 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.950926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/117d5a5e-d4a8-4274-9227-e6a687222bd6-console-serving-cert\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.952429 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.952408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-b589c8cc8-lzm92" event={"ID":"d44f95fb-f12a-415d-bd5d-affde1179526","Type":"ContainerStarted","Data":"dc7d28e517cfb01ff27cd571bd59f48dbeefbe2fad1cd2b9f13f1053c70b8388"}
Sep 15 16:33:28.952539 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.952525    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:33:28.956849 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.956830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcq7t\" (UniqueName: \"kubernetes.io/projected/117d5a5e-d4a8-4274-9227-e6a687222bd6-kube-api-access-hcq7t\") pod \"console-657d8c56dc-d5p7j\" (UID: \"117d5a5e-d4a8-4274-9227-e6a687222bd6\") " pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:28.971144 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:28.971096    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-b589c8cc8-lzm92" podStartSLOduration=1.18612359 podStartE2EDuration="1.971077928s" totalImagesPullingTime="784.954338ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:33:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:33:28.969029436 +0000 UTC m=+397.336854639" watchObservedRunningTime="2026-09-15 16:33:28.971077928 +0000 UTC m=+397.338903128"
Sep 15 16:33:29.047527 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:29.047432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:29.322125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:29.322102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-657d8c56dc-d5p7j"]
Sep 15 16:33:29.324284 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:33:29.324258    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod117d5a5e_d4a8_4274_9227_e6a687222bd6.slice/crio-78c12055cb264569d058f179bf99e6ca1900e510862376e31115819b0e5427eb WatchSource:0}: Error finding container 78c12055cb264569d058f179bf99e6ca1900e510862376e31115819b0e5427eb: Status 404 returned error can't find the container with id 78c12055cb264569d058f179bf99e6ca1900e510862376e31115819b0e5427eb
Sep 15 16:33:29.958833 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:29.958790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657d8c56dc-d5p7j" event={"ID":"117d5a5e-d4a8-4274-9227-e6a687222bd6","Type":"ContainerStarted","Data":"3e9d013fda25136c25ceda4d172844ae3f46f38b3104df91669745b1012e4045"}
Sep 15 16:33:29.959025 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:29.958852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-657d8c56dc-d5p7j" event={"ID":"117d5a5e-d4a8-4274-9227-e6a687222bd6","Type":"ContainerStarted","Data":"78c12055cb264569d058f179bf99e6ca1900e510862376e31115819b0e5427eb"}
Sep 15 16:33:29.978341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:29.978289    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-657d8c56dc-d5p7j" podStartSLOduration=1.978273527 podStartE2EDuration="1.978273527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:33:29.976946196 +0000 UTC m=+398.344771395" watchObservedRunningTime="2026-09-15 16:33:29.978273527 +0000 UTC m=+398.346098725"
Sep 15 16:33:39.048218 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:39.048180    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:39.048680 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:39.048240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:39.053226 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:39.053202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:40.006851 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:40.006820    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-657d8c56dc-d5p7j"
Sep 15 16:33:40.059537 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:40.059494    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b57f9b496-5scv8"]
Sep 15 16:33:59.964419 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:33:59.964386    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-b589c8cc8-lzm92"
Sep 15 16:34:01.265267 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.265227    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-25nm7"]
Sep 15 16:34:01.268841 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.268818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:01.271038 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.271015    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 15 16:34:01.271038 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.271032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-c62z9\""
Sep 15 16:34:01.279208 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.279174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-25nm7"]
Sep 15 16:34:01.307666 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.307633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-tls-certs\") pod \"model-serving-api-7b954ffc95-25nm7\" (UID: \"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd\") " pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:01.307826 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.307675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vht7\" (UniqueName: \"kubernetes.io/projected/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-kube-api-access-8vht7\") pod \"model-serving-api-7b954ffc95-25nm7\" (UID: \"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd\") " pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:01.408813 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.408773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vht7\" (UniqueName: \"kubernetes.io/projected/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-kube-api-access-8vht7\") pod \"model-serving-api-7b954ffc95-25nm7\" (UID: \"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd\") " pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:01.408981 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.408885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-tls-certs\") pod \"model-serving-api-7b954ffc95-25nm7\" (UID: \"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd\") " pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:01.409022 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:34:01.408983    2566 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 15 16:34:01.409061 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:34:01.409047    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-tls-certs podName:ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd nodeName:}" failed. No retries permitted until 2026-09-15 16:34:01.90902657 +0000 UTC m=+430.276851751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-tls-certs") pod "model-serving-api-7b954ffc95-25nm7" (UID: "ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd") : secret "model-serving-api-tls" not found
Sep 15 16:34:01.417326 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.417294    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vht7\" (UniqueName: \"kubernetes.io/projected/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-kube-api-access-8vht7\") pod \"model-serving-api-7b954ffc95-25nm7\" (UID: \"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd\") " pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:01.912860 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.912821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-tls-certs\") pod \"model-serving-api-7b954ffc95-25nm7\" (UID: \"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd\") " pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:01.915214 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:01.915182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd-tls-certs\") pod \"model-serving-api-7b954ffc95-25nm7\" (UID: \"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd\") " pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:02.182018 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:02.181931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:02.454438 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:02.454411    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-25nm7"]
Sep 15 16:34:02.456824 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:34:02.456797    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec79fddd_81d8_41dc_a6bd_a7f2dd8a2cdd.slice/crio-7b3ca3f180d02eb8faf848a56de360d0f8efab5644b9c319d64e6fda413cf733 WatchSource:0}: Error finding container 7b3ca3f180d02eb8faf848a56de360d0f8efab5644b9c319d64e6fda413cf733: Status 404 returned error can't find the container with id 7b3ca3f180d02eb8faf848a56de360d0f8efab5644b9c319d64e6fda413cf733
Sep 15 16:34:03.105539 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:03.105501    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-25nm7" event={"ID":"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd","Type":"ContainerStarted","Data":"7b3ca3f180d02eb8faf848a56de360d0f8efab5644b9c319d64e6fda413cf733"}
Sep 15 16:34:05.085542 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:05.085445    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b57f9b496-5scv8" podUID="4be1b87d-1653-4fbe-a8e8-7d3444aa8571" containerName="console" containerID="cri-o://a3e2c28782490d6985f9208f54ecf0c358a97fb74ed5efa439012aaf1176982d" gracePeriod=15
Sep 15 16:34:06.121103 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.121071    2566 generic.go:356] "Generic (PLEG): container finished" podID="4be1b87d-1653-4fbe-a8e8-7d3444aa8571" containerID="a3e2c28782490d6985f9208f54ecf0c358a97fb74ed5efa439012aaf1176982d" exitCode=0
Sep 15 16:34:06.121506 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.121143    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b57f9b496-5scv8" event={"ID":"4be1b87d-1653-4fbe-a8e8-7d3444aa8571","Type":"ContainerDied","Data":"a3e2c28782490d6985f9208f54ecf0c358a97fb74ed5efa439012aaf1176982d"}
Sep 15 16:34:06.267446 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.267420    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:34:06.351324 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351299    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-oauth-config\") pod \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") "
Sep 15 16:34:06.351433 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351356    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-684d6\" (UniqueName: \"kubernetes.io/projected/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-kube-api-access-684d6\") pod \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") "
Sep 15 16:34:06.351433 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351382    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-trusted-ca-bundle\") pod \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") "
Sep 15 16:34:06.351433 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351401    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-service-ca\") pod \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") "
Sep 15 16:34:06.351433 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-serving-cert\") pod \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") "
Sep 15 16:34:06.351661 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351450    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-config\") pod \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") "
Sep 15 16:34:06.351661 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351490    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-oauth-serving-cert\") pod \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\" (UID: \"4be1b87d-1653-4fbe-a8e8-7d3444aa8571\") "
Sep 15 16:34:06.351908 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351819    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-service-ca" (OuterVolumeSpecName: "service-ca") pod "4be1b87d-1653-4fbe-a8e8-7d3444aa8571" (UID: "4be1b87d-1653-4fbe-a8e8-7d3444aa8571"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:34:06.351908 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.351839    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4be1b87d-1653-4fbe-a8e8-7d3444aa8571" (UID: "4be1b87d-1653-4fbe-a8e8-7d3444aa8571"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:34:06.352077 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.352054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4be1b87d-1653-4fbe-a8e8-7d3444aa8571" (UID: "4be1b87d-1653-4fbe-a8e8-7d3444aa8571"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:34:06.352131 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.352080    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-config" (OuterVolumeSpecName: "console-config") pod "4be1b87d-1653-4fbe-a8e8-7d3444aa8571" (UID: "4be1b87d-1653-4fbe-a8e8-7d3444aa8571"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:34:06.353601 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.353535    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4be1b87d-1653-4fbe-a8e8-7d3444aa8571" (UID: "4be1b87d-1653-4fbe-a8e8-7d3444aa8571"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:34:06.353601 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.353554    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4be1b87d-1653-4fbe-a8e8-7d3444aa8571" (UID: "4be1b87d-1653-4fbe-a8e8-7d3444aa8571"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:34:06.353756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.353714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-kube-api-access-684d6" (OuterVolumeSpecName: "kube-api-access-684d6") pod "4be1b87d-1653-4fbe-a8e8-7d3444aa8571" (UID: "4be1b87d-1653-4fbe-a8e8-7d3444aa8571"). InnerVolumeSpecName "kube-api-access-684d6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:34:06.452258 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.452224    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-oauth-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:34:06.452258 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.452256    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-684d6\" (UniqueName: \"kubernetes.io/projected/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-kube-api-access-684d6\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:34:06.452258 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.452265    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-trusted-ca-bundle\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:34:06.452487 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.452274    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-service-ca\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:34:06.452487 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.452285    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-serving-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:34:06.452487 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.452293    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-console-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:34:06.452487 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:06.452301    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4be1b87d-1653-4fbe-a8e8-7d3444aa8571-oauth-serving-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:34:07.126700 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:07.126665    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b57f9b496-5scv8"
Sep 15 16:34:07.127138 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:07.126699    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b57f9b496-5scv8" event={"ID":"4be1b87d-1653-4fbe-a8e8-7d3444aa8571","Type":"ContainerDied","Data":"450e0139d4d2edd3905e75580b212a4b539b36f5a80ff984d9bf924e1dc456dc"}
Sep 15 16:34:07.127138 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:07.126760    2566 scope.go:117] "RemoveContainer" containerID="a3e2c28782490d6985f9208f54ecf0c358a97fb74ed5efa439012aaf1176982d"
Sep 15 16:34:07.128392 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:07.128362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-25nm7" event={"ID":"ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd","Type":"ContainerStarted","Data":"64e7674267c7268c3db26eaa9e699cc1a8391e523f4443c16b25652322279d4d"}
Sep 15 16:34:07.128510 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:07.128488    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:34:07.147613 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:07.147542    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-25nm7" podStartSLOduration=2.323088642 podStartE2EDuration="6.147527719s" totalImagesPullingTime="3.824439077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:34:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:34:07.146355462 +0000 UTC m=+435.514180662" watchObservedRunningTime="2026-09-15 16:34:07.147527719 +0000 UTC m=+435.515352918"
Sep 15 16:34:07.161875 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:07.161837    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b57f9b496-5scv8"]
Sep 15 16:34:07.166925 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:07.166893    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b57f9b496-5scv8"]
Sep 15 16:34:08.158704 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:08.158664    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be1b87d-1653-4fbe-a8e8-7d3444aa8571" path="/var/lib/kubelet/pods/4be1b87d-1653-4fbe-a8e8-7d3444aa8571/volumes"
Sep 15 16:34:18.138687 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:34:18.138615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-25nm7"
Sep 15 16:35:37.141320 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.141281    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"]
Sep 15 16:35:37.141842 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.141779    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be1b87d-1653-4fbe-a8e8-7d3444aa8571" containerName="console"
Sep 15 16:35:37.141842 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.141799    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be1b87d-1653-4fbe-a8e8-7d3444aa8571" containerName="console"
Sep 15 16:35:37.141962 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.141888    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be1b87d-1653-4fbe-a8e8-7d3444aa8571" containerName="console"
Sep 15 16:35:37.143849 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.143828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:37.146092 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.146068    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 15 16:35:37.152331 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.152310    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"]
Sep 15 16:35:37.284223 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.284188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b8154-d199-4482-9d70-f624576b894e-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9v668\" (UID: \"bc1b8154-d199-4482-9d70-f624576b894e\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:37.284223 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.284228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfkj\" (UniqueName: \"kubernetes.io/projected/bc1b8154-d199-4482-9d70-f624576b894e-kube-api-access-vlfkj\") pod \"seaweedfs-tls-custom-6644bf5d6c-9v668\" (UID: \"bc1b8154-d199-4482-9d70-f624576b894e\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:37.384872 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.384821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b8154-d199-4482-9d70-f624576b894e-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9v668\" (UID: \"bc1b8154-d199-4482-9d70-f624576b894e\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:37.384872 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.384873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfkj\" (UniqueName: \"kubernetes.io/projected/bc1b8154-d199-4482-9d70-f624576b894e-kube-api-access-vlfkj\") pod \"seaweedfs-tls-custom-6644bf5d6c-9v668\" (UID: \"bc1b8154-d199-4482-9d70-f624576b894e\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:37.385275 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.385249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b8154-d199-4482-9d70-f624576b894e-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-9v668\" (UID: \"bc1b8154-d199-4482-9d70-f624576b894e\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:37.393181 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.393112    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfkj\" (UniqueName: \"kubernetes.io/projected/bc1b8154-d199-4482-9d70-f624576b894e-kube-api-access-vlfkj\") pod \"seaweedfs-tls-custom-6644bf5d6c-9v668\" (UID: \"bc1b8154-d199-4482-9d70-f624576b894e\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:37.453958 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.453925    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:37.721710 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:37.721636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"]
Sep 15 16:35:37.725166 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:35:37.725137    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1b8154_d199_4482_9d70_f624576b894e.slice/crio-dee206b85f55da1985ff0a93c806d5ce2f5c3f798024e7e06586a7ed8b51a130 WatchSource:0}: Error finding container dee206b85f55da1985ff0a93c806d5ce2f5c3f798024e7e06586a7ed8b51a130: Status 404 returned error can't find the container with id dee206b85f55da1985ff0a93c806d5ce2f5c3f798024e7e06586a7ed8b51a130
Sep 15 16:35:38.525114 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:38.525081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668" event={"ID":"bc1b8154-d199-4482-9d70-f624576b894e","Type":"ContainerStarted","Data":"9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da"}
Sep 15 16:35:38.525458 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:38.525128    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668" event={"ID":"bc1b8154-d199-4482-9d70-f624576b894e","Type":"ContainerStarted","Data":"dee206b85f55da1985ff0a93c806d5ce2f5c3f798024e7e06586a7ed8b51a130"}
Sep 15 16:35:38.543284 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:38.543205    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668" podStartSLOduration=1.287419261 podStartE2EDuration="1.543185056s" totalImagesPullingTime="255.765795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:35:38.540691552 +0000 UTC m=+526.908516779" watchObservedRunningTime="2026-09-15 16:35:38.543185056 +0000 UTC m=+526.911010256"
Sep 15 16:35:39.794747 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:39.794710    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"]
Sep 15 16:35:40.534380 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:40.534320    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668" podUID="bc1b8154-d199-4482-9d70-f624576b894e" containerName="seaweedfs-tls-custom" containerID="cri-o://9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da" gracePeriod=30
Sep 15 16:35:41.773550 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:41.773520    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:41.921652 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:41.921621    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b8154-d199-4482-9d70-f624576b894e-data\") pod \"bc1b8154-d199-4482-9d70-f624576b894e\" (UID: \"bc1b8154-d199-4482-9d70-f624576b894e\") "
Sep 15 16:35:41.921811 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:41.921678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlfkj\" (UniqueName: \"kubernetes.io/projected/bc1b8154-d199-4482-9d70-f624576b894e-kube-api-access-vlfkj\") pod \"bc1b8154-d199-4482-9d70-f624576b894e\" (UID: \"bc1b8154-d199-4482-9d70-f624576b894e\") "
Sep 15 16:35:41.922947 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:41.922913    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1b8154-d199-4482-9d70-f624576b894e-data" (OuterVolumeSpecName: "data") pod "bc1b8154-d199-4482-9d70-f624576b894e" (UID: "bc1b8154-d199-4482-9d70-f624576b894e"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:35:41.923703 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:41.923679    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b8154-d199-4482-9d70-f624576b894e-kube-api-access-vlfkj" (OuterVolumeSpecName: "kube-api-access-vlfkj") pod "bc1b8154-d199-4482-9d70-f624576b894e" (UID: "bc1b8154-d199-4482-9d70-f624576b894e"). InnerVolumeSpecName "kube-api-access-vlfkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:35:42.022757 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.022720    2566 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc1b8154-d199-4482-9d70-f624576b894e-data\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:35:42.022757 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.022751    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlfkj\" (UniqueName: \"kubernetes.io/projected/bc1b8154-d199-4482-9d70-f624576b894e-kube-api-access-vlfkj\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:35:42.543322 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.543285    2566 generic.go:356] "Generic (PLEG): container finished" podID="bc1b8154-d199-4482-9d70-f624576b894e" containerID="9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da" exitCode=0
Sep 15 16:35:42.543466 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.543332    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"
Sep 15 16:35:42.543466 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.543344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668" event={"ID":"bc1b8154-d199-4482-9d70-f624576b894e","Type":"ContainerDied","Data":"9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da"}
Sep 15 16:35:42.543466 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.543391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-9v668" event={"ID":"bc1b8154-d199-4482-9d70-f624576b894e","Type":"ContainerDied","Data":"dee206b85f55da1985ff0a93c806d5ce2f5c3f798024e7e06586a7ed8b51a130"}
Sep 15 16:35:42.543466 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.543419    2566 scope.go:117] "RemoveContainer" containerID="9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da"
Sep 15 16:35:42.554173 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.554151    2566 scope.go:117] "RemoveContainer" containerID="9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da"
Sep 15 16:35:42.554447 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:35:42.554425    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da\": container with ID starting with 9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da not found: ID does not exist" containerID="9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da"
Sep 15 16:35:42.554514 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.554456    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da"} err="failed to get container status \"9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da\": rpc error: code = NotFound desc = could not find container \"9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da\": container with ID starting with 9a950ff514216969e67222198f61d509c9d64dbd9fa2fa87dc10841b75a9e6da not found: ID does not exist"
Sep 15 16:35:42.562713 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.562685    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"]
Sep 15 16:35:42.565671 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.565651    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-9v668"]
Sep 15 16:35:42.590835 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.590815    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-wwm86"]
Sep 15 16:35:42.591127 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.591115    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1b8154-d199-4482-9d70-f624576b894e" containerName="seaweedfs-tls-custom"
Sep 15 16:35:42.591168 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.591129    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1b8154-d199-4482-9d70-f624576b894e" containerName="seaweedfs-tls-custom"
Sep 15 16:35:42.591216 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.591186    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1b8154-d199-4482-9d70-f624576b894e" containerName="seaweedfs-tls-custom"
Sep 15 16:35:42.594170 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.594153    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.596417 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.596390    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 15 16:35:42.596519 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.596471    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 15 16:35:42.603375 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.603354    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-wwm86"]
Sep 15 16:35:42.729115 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.729076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj5cw\" (UniqueName: \"kubernetes.io/projected/97ec9d36-dc40-403d-9242-ee2b8ff687fc-kube-api-access-bj5cw\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.729284 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.729120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/97ec9d36-dc40-403d-9242-ee2b8ff687fc-data\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.729284 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.729247    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/97ec9d36-dc40-403d-9242-ee2b8ff687fc-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.830425 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.830343    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj5cw\" (UniqueName: \"kubernetes.io/projected/97ec9d36-dc40-403d-9242-ee2b8ff687fc-kube-api-access-bj5cw\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.830425 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.830381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/97ec9d36-dc40-403d-9242-ee2b8ff687fc-data\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.830928 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.830508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/97ec9d36-dc40-403d-9242-ee2b8ff687fc-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.830928 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.830735    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/97ec9d36-dc40-403d-9242-ee2b8ff687fc-data\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.832932 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.832913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/97ec9d36-dc40-403d-9242-ee2b8ff687fc-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.839249 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.839229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj5cw\" (UniqueName: \"kubernetes.io/projected/97ec9d36-dc40-403d-9242-ee2b8ff687fc-kube-api-access-bj5cw\") pod \"seaweedfs-tls-custom-8b44d767d-wwm86\" (UID: \"97ec9d36-dc40-403d-9242-ee2b8ff687fc\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:42.903398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:42.903369    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86"
Sep 15 16:35:43.172254 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:43.172230    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-wwm86"]
Sep 15 16:35:43.174923 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:35:43.174890    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ec9d36_dc40_403d_9242_ee2b8ff687fc.slice/crio-c08bb5f98e87d5f87b1ce932896a85e3e18a9f3f07c0fa6dec8335b3fd3d1c50 WatchSource:0}: Error finding container c08bb5f98e87d5f87b1ce932896a85e3e18a9f3f07c0fa6dec8335b3fd3d1c50: Status 404 returned error can't find the container with id c08bb5f98e87d5f87b1ce932896a85e3e18a9f3f07c0fa6dec8335b3fd3d1c50
Sep 15 16:35:43.550970 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:43.550935    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86" event={"ID":"97ec9d36-dc40-403d-9242-ee2b8ff687fc","Type":"ContainerStarted","Data":"bf2e3ac65118258eb3ac236f86f5f93f14f80718be1b7898e22da861b62f04c3"}
Sep 15 16:35:43.551118 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:43.551027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86" event={"ID":"97ec9d36-dc40-403d-9242-ee2b8ff687fc","Type":"ContainerStarted","Data":"c08bb5f98e87d5f87b1ce932896a85e3e18a9f3f07c0fa6dec8335b3fd3d1c50"}
Sep 15 16:35:43.568709 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:43.568655    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-wwm86" podStartSLOduration=1.274364153 podStartE2EDuration="1.568638289s" totalImagesPullingTime="294.274136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:35:43.56619936 +0000 UTC m=+531.934024560" watchObservedRunningTime="2026-09-15 16:35:43.568638289 +0000 UTC m=+531.936463487"
Sep 15 16:35:44.159903 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:35:44.159806    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1b8154-d199-4482-9d70-f624576b894e" path="/var/lib/kubelet/pods/bc1b8154-d199-4482-9d70-f624576b894e/volumes"
Sep 15 16:36:52.064449 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:36:52.064421    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:36:52.065496 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:36:52.065471    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:41:52.089861 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:52.089785    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:41:52.093061 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:52.093038    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:41:55.475874 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.475834    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"]
Sep 15 16:41:55.479435 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.479413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.482075 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.482046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:41:55.482219 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.482076    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:41:55.482219 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.482168    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:41:55.482986 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.482967    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:41:55.483092 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.482987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 15 16:41:55.483092 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.483025    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 15 16:41:55.483203 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.483096    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:41:55.489843 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.489821    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"]
Sep 15 16:41:55.658450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.658417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d7fa082-2303-4110-b7e2-3c647ae15f90-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.658450 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.658453    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kk2v\" (UniqueName: \"kubernetes.io/projected/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-api-access-7kk2v\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.658686 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.658537    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.658686 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.658559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.658686 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.658632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d7fa082-2303-4110-b7e2-3c647ae15f90-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.759414 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.759338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d7fa082-2303-4110-b7e2-3c647ae15f90-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.759414 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.759393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d7fa082-2303-4110-b7e2-3c647ae15f90-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.759630 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.759421    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kk2v\" (UniqueName: \"kubernetes.io/projected/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-api-access-7kk2v\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.759630 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.759467    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.759630 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.759493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.759794 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.759774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d7fa082-2303-4110-b7e2-3c647ae15f90-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.760176 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.760145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.760267 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.760247    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.761952 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.761931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d7fa082-2303-4110-b7e2-3c647ae15f90-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.767495 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.767464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kk2v\" (UniqueName: \"kubernetes.io/projected/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-api-access-7kk2v\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:55.790410 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:55.790385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:41:56.063028 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:56.063003    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"]
Sep 15 16:41:56.065343 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:41:56.065318    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7fa082_2303_4110_b7e2_3c647ae15f90.slice/crio-d6425c3b83567744f0744a69b633917201a3526ddecac23af4b5ce38d5012735 WatchSource:0}: Error finding container d6425c3b83567744f0744a69b633917201a3526ddecac23af4b5ce38d5012735: Status 404 returned error can't find the container with id d6425c3b83567744f0744a69b633917201a3526ddecac23af4b5ce38d5012735
Sep 15 16:41:56.067307 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:56.067286    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:41:56.163164 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:41:56.163132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" event={"ID":"5d7fa082-2303-4110-b7e2-3c647ae15f90","Type":"ContainerStarted","Data":"d6425c3b83567744f0744a69b633917201a3526ddecac23af4b5ce38d5012735"}
Sep 15 16:42:00.190515 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:00.190479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" event={"ID":"5d7fa082-2303-4110-b7e2-3c647ae15f90","Type":"ContainerStarted","Data":"c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3"}
Sep 15 16:42:04.208690 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:04.208649    2566 generic.go:356] "Generic (PLEG): container finished" podID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerID="c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3" exitCode=0
Sep 15 16:42:04.209070 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:04.208719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" event={"ID":"5d7fa082-2303-4110-b7e2-3c647ae15f90","Type":"ContainerDied","Data":"c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3"}
Sep 15 16:42:27.342523 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:27.342484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" event={"ID":"5d7fa082-2303-4110-b7e2-3c647ae15f90","Type":"ContainerStarted","Data":"0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1"}
Sep 15 16:42:29.354529 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:29.354439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" event={"ID":"5d7fa082-2303-4110-b7e2-3c647ae15f90","Type":"ContainerStarted","Data":"ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba"}
Sep 15 16:42:29.354966 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:29.354678    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:42:29.354966 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:29.354723    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:42:29.374596 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:29.374517    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" podStartSLOduration=2.329883579 podStartE2EDuration="34.374499745s" totalImagesPullingTime="28.197408696s" totalInitContainerRuntime="3.84720747s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:41:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:42:29.372463648 +0000 UTC m=+937.740288848" watchObservedRunningTime="2026-09-15 16:42:29.374499745 +0000 UTC m=+937.742324944"
Sep 15 16:42:35.364826 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:42:35.364795    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:43:05.365762 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:05.365723    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 15 16:43:15.366300 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:15.366203    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 15 16:43:25.365332 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:25.365286    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 15 16:43:35.365507 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:35.365471    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.132.0.36:8080: connect: connection refused"
Sep 15 16:43:45.460740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:45.460702    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:43:55.610531 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:55.609749    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"]
Sep 15 16:43:55.610531 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:55.610154    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kserve-container" containerID="cri-o://0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1" gracePeriod=30
Sep 15 16:43:55.610531 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:55.610385    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kube-rbac-proxy" containerID="cri-o://ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba" gracePeriod=30
Sep 15 16:43:55.736211 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:55.736183    2566 generic.go:356] "Generic (PLEG): container finished" podID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerID="ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba" exitCode=2
Sep 15 16:43:55.736338 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:43:55.736254    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" event={"ID":"5d7fa082-2303-4110-b7e2-3c647ae15f90","Type":"ContainerDied","Data":"ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba"}
Sep 15 16:44:00.061714 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.061687    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:44:00.150544 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.150512    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d7fa082-2303-4110-b7e2-3c647ae15f90-proxy-tls\") pod \"5d7fa082-2303-4110-b7e2-3c647ae15f90\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") "
Sep 15 16:44:00.150724 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.150584    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-cabundle-cert\") pod \"5d7fa082-2303-4110-b7e2-3c647ae15f90\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") "
Sep 15 16:44:00.150724 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.150613    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d7fa082-2303-4110-b7e2-3c647ae15f90-kserve-provision-location\") pod \"5d7fa082-2303-4110-b7e2-3c647ae15f90\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") "
Sep 15 16:44:00.150724 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.150651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-rbac-proxy-sar-config\") pod \"5d7fa082-2303-4110-b7e2-3c647ae15f90\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") "
Sep 15 16:44:00.150724 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.150669    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kk2v\" (UniqueName: \"kubernetes.io/projected/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-api-access-7kk2v\") pod \"5d7fa082-2303-4110-b7e2-3c647ae15f90\" (UID: \"5d7fa082-2303-4110-b7e2-3c647ae15f90\") "
Sep 15 16:44:00.150997 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.150961    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d7fa082-2303-4110-b7e2-3c647ae15f90-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d7fa082-2303-4110-b7e2-3c647ae15f90" (UID: "5d7fa082-2303-4110-b7e2-3c647ae15f90"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:44:00.150997 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.150985    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5d7fa082-2303-4110-b7e2-3c647ae15f90" (UID: "5d7fa082-2303-4110-b7e2-3c647ae15f90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:00.151133 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.150971    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d7fa082-2303-4110-b7e2-3c647ae15f90" (UID: "5d7fa082-2303-4110-b7e2-3c647ae15f90"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:44:00.152846 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.152823    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-api-access-7kk2v" (OuterVolumeSpecName: "kube-api-access-7kk2v") pod "5d7fa082-2303-4110-b7e2-3c647ae15f90" (UID: "5d7fa082-2303-4110-b7e2-3c647ae15f90"). InnerVolumeSpecName "kube-api-access-7kk2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:44:00.152915 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.152883    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7fa082-2303-4110-b7e2-3c647ae15f90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5d7fa082-2303-4110-b7e2-3c647ae15f90" (UID: "5d7fa082-2303-4110-b7e2-3c647ae15f90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:44:00.251894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.251860    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5d7fa082-2303-4110-b7e2-3c647ae15f90-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:44:00.251894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.251889    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:44:00.251894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.251899    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d7fa082-2303-4110-b7e2-3c647ae15f90-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:44:00.252114 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.251977    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:44:00.252114 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.251989    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kk2v\" (UniqueName: \"kubernetes.io/projected/5d7fa082-2303-4110-b7e2-3c647ae15f90-kube-api-access-7kk2v\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:44:00.759920 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.759886    2566 generic.go:356] "Generic (PLEG): container finished" podID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerID="0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1" exitCode=0
Sep 15 16:44:00.760080 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.759943    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"
Sep 15 16:44:00.760080 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.759962    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" event={"ID":"5d7fa082-2303-4110-b7e2-3c647ae15f90","Type":"ContainerDied","Data":"0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1"}
Sep 15 16:44:00.760080 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.760022    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl" event={"ID":"5d7fa082-2303-4110-b7e2-3c647ae15f90","Type":"ContainerDied","Data":"d6425c3b83567744f0744a69b633917201a3526ddecac23af4b5ce38d5012735"}
Sep 15 16:44:00.760080 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.760048    2566 scope.go:117] "RemoveContainer" containerID="ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba"
Sep 15 16:44:00.768795 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.768774    2566 scope.go:117] "RemoveContainer" containerID="0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1"
Sep 15 16:44:00.772348 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772326    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"]
Sep 15 16:44:00.772705 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772691    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="storage-initializer"
Sep 15 16:44:00.772751 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772708    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="storage-initializer"
Sep 15 16:44:00.772751 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772732    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kube-rbac-proxy"
Sep 15 16:44:00.772751 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772742    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kube-rbac-proxy"
Sep 15 16:44:00.772846 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772761    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kserve-container"
Sep 15 16:44:00.772846 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772770    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kserve-container"
Sep 15 16:44:00.772846 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772840    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kserve-container"
Sep 15 16:44:00.772939 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.772851    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" containerName="kube-rbac-proxy"
Sep 15 16:44:00.775784 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.775768    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.776715 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.776694    2566 scope.go:117] "RemoveContainer" containerID="c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3"
Sep 15 16:44:00.777985 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.777961    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:44:00.777985 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.777976    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 15 16:44:00.778127 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.778004    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Sep 15 16:44:00.778267 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.778251    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:44:00.778335 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.778321    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:44:00.778390 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.778350    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:44:00.778445 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.778381    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:44:00.781189 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.781166    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"]
Sep 15 16:44:00.784957 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.784345    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"]
Sep 15 16:44:00.787148 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.787124    2566 scope.go:117] "RemoveContainer" containerID="ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba"
Sep 15 16:44:00.787317 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.787290    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-cjxzl"]
Sep 15 16:44:00.787417 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:44:00.787401    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba\": container with ID starting with ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba not found: ID does not exist" containerID="ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba"
Sep 15 16:44:00.787474 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.787424    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba"} err="failed to get container status \"ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba\": rpc error: code = NotFound desc = could not find container \"ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba\": container with ID starting with ffc2e9a237a3900b725ec91d4462ed56bc4ca220f5d4b66647612085f6e236ba not found: ID does not exist"
Sep 15 16:44:00.787474 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.787440    2566 scope.go:117] "RemoveContainer" containerID="0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1"
Sep 15 16:44:00.787757 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:44:00.787731    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1\": container with ID starting with 0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1 not found: ID does not exist" containerID="0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1"
Sep 15 16:44:00.787859 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.787762    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1"} err="failed to get container status \"0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1\": rpc error: code = NotFound desc = could not find container \"0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1\": container with ID starting with 0087c83a361fcdd145a9e6ae3a66f880dd5735ed8cf471743daabe298c36d6f1 not found: ID does not exist"
Sep 15 16:44:00.787859 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.787785    2566 scope.go:117] "RemoveContainer" containerID="c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3"
Sep 15 16:44:00.788113 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:44:00.788092    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3\": container with ID starting with c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3 not found: ID does not exist" containerID="c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3"
Sep 15 16:44:00.788113 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.788116    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3"} err="failed to get container status \"c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3\": rpc error: code = NotFound desc = could not find container \"c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3\": container with ID starting with c74d94d96592fd36f19a2cdb3f86bf644b434d3cdb3b39a8bbd669b6cf7567b3 not found: ID does not exist"
Sep 15 16:44:00.857465 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.857433    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.857465 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.857468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thwc\" (UniqueName: \"kubernetes.io/projected/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-api-access-5thwc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.857706 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.857487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.857706 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.857616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590cd7d-c040-43ee-bf32-2913d09bb654-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.857706 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.857658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1590cd7d-c040-43ee-bf32-2913d09bb654-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.958804 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.958775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.958947 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.958811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thwc\" (UniqueName: \"kubernetes.io/projected/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-api-access-5thwc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.958947 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.958830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.958947 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.958865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590cd7d-c040-43ee-bf32-2913d09bb654-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.958947 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.958895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1590cd7d-c040-43ee-bf32-2913d09bb654-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.959321 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.959297    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1590cd7d-c040-43ee-bf32-2913d09bb654-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.959532 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.959512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.959608 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.959537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.961205 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.961187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590cd7d-c040-43ee-bf32-2913d09bb654-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:00.966897 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:00.966877    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thwc\" (UniqueName: \"kubernetes.io/projected/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-api-access-5thwc\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:01.088851 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:01.088781    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:01.358829 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:01.358751    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"]
Sep 15 16:44:01.361230 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:44:01.361201    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1590cd7d_c040_43ee_bf32_2913d09bb654.slice/crio-e0b247dfa2b8f720113fb93462560d780a27eb5b3d59cd5330c48286dd5c8c09 WatchSource:0}: Error finding container e0b247dfa2b8f720113fb93462560d780a27eb5b3d59cd5330c48286dd5c8c09: Status 404 returned error can't find the container with id e0b247dfa2b8f720113fb93462560d780a27eb5b3d59cd5330c48286dd5c8c09
Sep 15 16:44:01.766205 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:01.766170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" event={"ID":"1590cd7d-c040-43ee-bf32-2913d09bb654","Type":"ContainerStarted","Data":"17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314"}
Sep 15 16:44:01.766378 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:01.766240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" event={"ID":"1590cd7d-c040-43ee-bf32-2913d09bb654","Type":"ContainerStarted","Data":"e0b247dfa2b8f720113fb93462560d780a27eb5b3d59cd5330c48286dd5c8c09"}
Sep 15 16:44:02.158436 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:02.158398    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7fa082-2303-4110-b7e2-3c647ae15f90" path="/var/lib/kubelet/pods/5d7fa082-2303-4110-b7e2-3c647ae15f90/volumes"
Sep 15 16:44:05.785673 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:05.785641    2566 generic.go:356] "Generic (PLEG): container finished" podID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerID="17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314" exitCode=0
Sep 15 16:44:05.786113 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:05.785716    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" event={"ID":"1590cd7d-c040-43ee-bf32-2913d09bb654","Type":"ContainerDied","Data":"17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314"}
Sep 15 16:44:06.793543 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:06.793506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" event={"ID":"1590cd7d-c040-43ee-bf32-2913d09bb654","Type":"ContainerStarted","Data":"cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7"}
Sep 15 16:44:06.794010 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:06.793600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" event={"ID":"1590cd7d-c040-43ee-bf32-2913d09bb654","Type":"ContainerStarted","Data":"b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd"}
Sep 15 16:44:06.794010 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:06.793826    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:06.813225 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:06.813179    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podStartSLOduration=3.053118682 podStartE2EDuration="6.813166704s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.760048022s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:44:06.811839502 +0000 UTC m=+1035.179664704" watchObservedRunningTime="2026-09-15 16:44:06.813166704 +0000 UTC m=+1035.180991903"
Sep 15 16:44:07.798735 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:07.798703    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:13.808585 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:13.808547    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:44:43.809630 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:43.809583    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 15 16:44:53.809512 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:44:53.809472    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 15 16:45:03.809112 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:03.809068    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 15 16:45:13.809713 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:13.809669    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 15 16:45:22.260166 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:22.260129    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:45:30.829524 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:30.829480    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"]
Sep 15 16:45:30.830030 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:30.829803    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container" containerID="cri-o://b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd" gracePeriod=30
Sep 15 16:45:30.830030 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:30.829898    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kube-rbac-proxy" containerID="cri-o://cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7" gracePeriod=30
Sep 15 16:45:31.176669 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:31.176632    2566 generic.go:356] "Generic (PLEG): container finished" podID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerID="cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7" exitCode=2
Sep 15 16:45:31.176833 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:31.176702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" event={"ID":"1590cd7d-c040-43ee-bf32-2913d09bb654","Type":"ContainerDied","Data":"cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7"}
Sep 15 16:45:32.155867 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:32.155816    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.37:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.132.0.37:8080: connect: connection refused"
Sep 15 16:45:33.804564 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:33.804521    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Sep 15 16:45:35.173088 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.173065    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:45:35.196271 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.196236    2566 generic.go:356] "Generic (PLEG): container finished" podID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerID="b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd" exitCode=0
Sep 15 16:45:35.196414 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.196287    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"
Sep 15 16:45:35.196414 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.196308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" event={"ID":"1590cd7d-c040-43ee-bf32-2913d09bb654","Type":"ContainerDied","Data":"b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd"}
Sep 15 16:45:35.196414 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.196366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj" event={"ID":"1590cd7d-c040-43ee-bf32-2913d09bb654","Type":"ContainerDied","Data":"e0b247dfa2b8f720113fb93462560d780a27eb5b3d59cd5330c48286dd5c8c09"}
Sep 15 16:45:35.196414 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.196402    2566 scope.go:117] "RemoveContainer" containerID="cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7"
Sep 15 16:45:35.206313 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.206290    2566 scope.go:117] "RemoveContainer" containerID="b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd"
Sep 15 16:45:35.215644 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.215624    2566 scope.go:117] "RemoveContainer" containerID="17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314"
Sep 15 16:45:35.222824 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.222794    2566 scope.go:117] "RemoveContainer" containerID="cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7"
Sep 15 16:45:35.223097 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:45:35.223071    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7\": container with ID starting with cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7 not found: ID does not exist" containerID="cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7"
Sep 15 16:45:35.223182 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.223102    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7"} err="failed to get container status \"cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7\": rpc error: code = NotFound desc = could not find container \"cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7\": container with ID starting with cdf609124eec6dbcef07cccd9b4be7ec3d2e313a2ba8bae31840824d53db83a7 not found: ID does not exist"
Sep 15 16:45:35.223182 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.223118    2566 scope.go:117] "RemoveContainer" containerID="b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd"
Sep 15 16:45:35.223357 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:45:35.223338    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd\": container with ID starting with b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd not found: ID does not exist" containerID="b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd"
Sep 15 16:45:35.223422 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.223366    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd"} err="failed to get container status \"b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd\": rpc error: code = NotFound desc = could not find container \"b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd\": container with ID starting with b65c9654458db76fa2d44834a1c4ad2769bf86167172470bdf42d2b32af577fd not found: ID does not exist"
Sep 15 16:45:35.223422 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.223388    2566 scope.go:117] "RemoveContainer" containerID="17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314"
Sep 15 16:45:35.223681 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:45:35.223656    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314\": container with ID starting with 17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314 not found: ID does not exist" containerID="17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314"
Sep 15 16:45:35.223734 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.223692    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314"} err="failed to get container status \"17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314\": rpc error: code = NotFound desc = could not find container \"17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314\": container with ID starting with 17c5f590620b2a57944fd689ef98eb9ae2b01eccc055fd49989cc5949a178314 not found: ID does not exist"
Sep 15 16:45:35.307610 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.307558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1590cd7d-c040-43ee-bf32-2913d09bb654-kserve-provision-location\") pod \"1590cd7d-c040-43ee-bf32-2913d09bb654\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") "
Sep 15 16:45:35.307799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.307640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-cabundle-cert\") pod \"1590cd7d-c040-43ee-bf32-2913d09bb654\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") "
Sep 15 16:45:35.307799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.307677    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-rbac-proxy-sar-config\") pod \"1590cd7d-c040-43ee-bf32-2913d09bb654\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") "
Sep 15 16:45:35.307799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.307702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590cd7d-c040-43ee-bf32-2913d09bb654-proxy-tls\") pod \"1590cd7d-c040-43ee-bf32-2913d09bb654\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") "
Sep 15 16:45:35.307799 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.307737    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5thwc\" (UniqueName: \"kubernetes.io/projected/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-api-access-5thwc\") pod \"1590cd7d-c040-43ee-bf32-2913d09bb654\" (UID: \"1590cd7d-c040-43ee-bf32-2913d09bb654\") "
Sep 15 16:45:35.308028 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.307911    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1590cd7d-c040-43ee-bf32-2913d09bb654-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1590cd7d-c040-43ee-bf32-2913d09bb654" (UID: "1590cd7d-c040-43ee-bf32-2913d09bb654"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:45:35.308028 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.307996    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1590cd7d-c040-43ee-bf32-2913d09bb654" (UID: "1590cd7d-c040-43ee-bf32-2913d09bb654"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:45:35.308133 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.308067    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1590cd7d-c040-43ee-bf32-2913d09bb654" (UID: "1590cd7d-c040-43ee-bf32-2913d09bb654"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:45:35.308211 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.308190    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1590cd7d-c040-43ee-bf32-2913d09bb654-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:45:35.309908 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.309877    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1590cd7d-c040-43ee-bf32-2913d09bb654-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1590cd7d-c040-43ee-bf32-2913d09bb654" (UID: "1590cd7d-c040-43ee-bf32-2913d09bb654"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:45:35.310022 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.310002    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-api-access-5thwc" (OuterVolumeSpecName: "kube-api-access-5thwc") pod "1590cd7d-c040-43ee-bf32-2913d09bb654" (UID: "1590cd7d-c040-43ee-bf32-2913d09bb654"). InnerVolumeSpecName "kube-api-access-5thwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:45:35.409512 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.409477    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thwc\" (UniqueName: \"kubernetes.io/projected/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-api-access-5thwc\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:45:35.409512 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.409511    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:45:35.409512 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.409521    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1590cd7d-c040-43ee-bf32-2913d09bb654-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:45:35.409705 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.409531    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1590cd7d-c040-43ee-bf32-2913d09bb654-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:45:35.518224 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.518198    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"]
Sep 15 16:45:35.523619 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:35.523600    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-db7nj"]
Sep 15 16:45:36.013584 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.013540    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"]
Sep 15 16:45:36.013927 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.013914    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kube-rbac-proxy"
Sep 15 16:45:36.013976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.013929    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kube-rbac-proxy"
Sep 15 16:45:36.013976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.013945    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container"
Sep 15 16:45:36.013976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.013952    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container"
Sep 15 16:45:36.013976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.013966    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="storage-initializer"
Sep 15 16:45:36.013976 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.013972    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="storage-initializer"
Sep 15 16:45:36.014119 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.014019    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kserve-container"
Sep 15 16:45:36.014119 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.014029    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" containerName="kube-rbac-proxy"
Sep 15 16:45:36.017190 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.017169    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.019622 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.019589    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:45:36.019721 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.019596    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:45:36.019721 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.019632    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:45:36.020621 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.020603    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Sep 15 16:45:36.020728 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.020631    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Sep 15 16:45:36.020728 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.020637    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:45:36.020728 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.020633    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:45:36.026871 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.026851    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"]
Sep 15 16:45:36.113546 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.113520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.113701 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.113554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlbbf\" (UniqueName: \"kubernetes.io/projected/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-api-access-hlbbf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.113701 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.113604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.113701 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.113662    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.113855 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.113773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.159705 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.159679    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1590cd7d-c040-43ee-bf32-2913d09bb654" path="/var/lib/kubelet/pods/1590cd7d-c040-43ee-bf32-2913d09bb654/volumes"
Sep 15 16:45:36.214686 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.214659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.215098 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.214704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.215098 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:45:36.214798    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 15 16:45:36.215098 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.214815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlbbf\" (UniqueName: \"kubernetes.io/projected/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-api-access-hlbbf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.215098 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.214851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.215098 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:45:36.214869    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls podName:cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254 nodeName:}" failed. No retries permitted until 2026-09-15 16:45:36.714848088 +0000 UTC m=+1125.082673282 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" (UID: "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Sep 15 16:45:36.215098 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.215054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.215341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.215232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.215341 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.215262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.215564 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.215545    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.225016 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.224995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlbbf\" (UniqueName: \"kubernetes.io/projected/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-api-access-hlbbf\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.719248 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.719214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.721562 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.721531    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:36.928506 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:36.928472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:37.199329 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:37.199304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"]
Sep 15 16:45:37.201228 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:45:37.201203    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6b4c17_e67c_42a2_a2c1_4e4ef9ffb254.slice/crio-ad07bf80d23c1613ea67ab68089f21d74366f694de228245b3f11c7776895564 WatchSource:0}: Error finding container ad07bf80d23c1613ea67ab68089f21d74366f694de228245b3f11c7776895564: Status 404 returned error can't find the container with id ad07bf80d23c1613ea67ab68089f21d74366f694de228245b3f11c7776895564
Sep 15 16:45:37.207532 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:37.207505    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" event={"ID":"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254","Type":"ContainerStarted","Data":"ad07bf80d23c1613ea67ab68089f21d74366f694de228245b3f11c7776895564"}
Sep 15 16:45:38.213451 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:38.213413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" event={"ID":"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254","Type":"ContainerStarted","Data":"30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131"}
Sep 15 16:45:41.227885 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:41.227801    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerID="30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131" exitCode=0
Sep 15 16:45:41.227885 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:41.227854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" event={"ID":"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254","Type":"ContainerDied","Data":"30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131"}
Sep 15 16:45:42.235116 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:42.235082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" event={"ID":"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254","Type":"ContainerStarted","Data":"0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140"}
Sep 15 16:45:42.235606 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:42.235158    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" event={"ID":"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254","Type":"ContainerStarted","Data":"403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742"}
Sep 15 16:45:42.235606 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:42.235374    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:42.255125 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:42.255073    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podStartSLOduration=3.568733281 podStartE2EDuration="7.25505695s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.686323669s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:45:42.253383515 +0000 UTC m=+1130.621208739" watchObservedRunningTime="2026-09-15 16:45:42.25505695 +0000 UTC m=+1130.622882175"
Sep 15 16:45:43.240869 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:43.240838    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:45:49.250686 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:45:49.250655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:46:19.252254 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:46:19.252162    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 15 16:46:29.252012 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:46:29.251968    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 15 16:46:39.251536 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:46:39.251495    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 15 16:46:49.251797 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:46:49.251754    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 15 16:46:52.117406 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:46:52.117379    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:46:52.121309 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:46:52.121283    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:46:59.268426 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:46:59.268393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:47:06.074665 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:06.074631    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"]
Sep 15 16:47:06.075145 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:06.074881    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container" containerID="cri-o://403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742" gracePeriod=30
Sep 15 16:47:06.075145 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:06.074912    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kube-rbac-proxy" containerID="cri-o://0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140" gracePeriod=30
Sep 15 16:47:06.615398 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:06.615366    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerID="0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140" exitCode=2
Sep 15 16:47:06.615552 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:06.615410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" event={"ID":"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254","Type":"ContainerDied","Data":"0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140"}
Sep 15 16:47:09.245905 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:09.245863    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Sep 15 16:47:09.251827 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:09.251798    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.38:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.132.0.38:8080: connect: connection refused"
Sep 15 16:47:10.530356 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.530330    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:47:10.548812 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.548788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kserve-provision-location\") pod \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") "
Sep 15 16:47:10.548929 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.548839    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls\") pod \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") "
Sep 15 16:47:10.548929 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.548899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-rbac-proxy-sar-config\") pod \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") "
Sep 15 16:47:10.548929 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.548924    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlbbf\" (UniqueName: \"kubernetes.io/projected/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-api-access-hlbbf\") pod \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") "
Sep 15 16:47:10.549085 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.549063    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-cabundle-cert\") pod \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\" (UID: \"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254\") "
Sep 15 16:47:10.549137 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.549085    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" (UID: "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:47:10.549233 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.549212    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" (UID: "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:47:10.549383 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.549358    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:47:10.549383 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.549376    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:47:10.549530 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.549407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" (UID: "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:47:10.550990 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.550972    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-api-access-hlbbf" (OuterVolumeSpecName: "kube-api-access-hlbbf") pod "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" (UID: "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254"). InnerVolumeSpecName "kube-api-access-hlbbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:47:10.551043 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.550973    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" (UID: "cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:47:10.635365 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.635339    2566 generic.go:356] "Generic (PLEG): container finished" podID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerID="403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742" exitCode=0
Sep 15 16:47:10.635501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.635404    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" event={"ID":"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254","Type":"ContainerDied","Data":"403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742"}
Sep 15 16:47:10.635501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.635464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh" event={"ID":"cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254","Type":"ContainerDied","Data":"ad07bf80d23c1613ea67ab68089f21d74366f694de228245b3f11c7776895564"}
Sep 15 16:47:10.635501 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.635413    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"
Sep 15 16:47:10.635678 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.635490    2566 scope.go:117] "RemoveContainer" containerID="0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140"
Sep 15 16:47:10.643889 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.643875    2566 scope.go:117] "RemoveContainer" containerID="403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742"
Sep 15 16:47:10.652046 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.652022    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:47:10.652046 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.652046    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlbbf\" (UniqueName: \"kubernetes.io/projected/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-kube-api-access-hlbbf\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:47:10.652192 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.652057    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:47:10.653489 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.653471    2566 scope.go:117] "RemoveContainer" containerID="30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131"
Sep 15 16:47:10.657927 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.657907    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"]
Sep 15 16:47:10.661053 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.661038    2566 scope.go:117] "RemoveContainer" containerID="0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140"
Sep 15 16:47:10.661304 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:47:10.661277    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140\": container with ID starting with 0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140 not found: ID does not exist" containerID="0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140"
Sep 15 16:47:10.661393 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.661304    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140"} err="failed to get container status \"0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140\": rpc error: code = NotFound desc = could not find container \"0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140\": container with ID starting with 0698284524ef211419ec409cbbc86c96d994dc78bbcde07dfa58b3a7fdd3c140 not found: ID does not exist"
Sep 15 16:47:10.661393 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.661320    2566 scope.go:117] "RemoveContainer" containerID="403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742"
Sep 15 16:47:10.661649 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:47:10.661613    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742\": container with ID starting with 403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742 not found: ID does not exist" containerID="403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742"
Sep 15 16:47:10.661750 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.661643    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742"} err="failed to get container status \"403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742\": rpc error: code = NotFound desc = could not find container \"403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742\": container with ID starting with 403cfbb2f6e19897536c8cf432196eb76cfbed8b21334fbc8dc6d666bcbb3742 not found: ID does not exist"
Sep 15 16:47:10.661750 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.661661    2566 scope.go:117] "RemoveContainer" containerID="30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131"
Sep 15 16:47:10.662001 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:47:10.661974    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131\": container with ID starting with 30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131 not found: ID does not exist" containerID="30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131"
Sep 15 16:47:10.662091 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.662008    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131"} err="failed to get container status \"30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131\": rpc error: code = NotFound desc = could not find container \"30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131\": container with ID starting with 30d60926af092b38cf88da36370a6ec888e1137455d09cb322f3534316aaf131 not found: ID does not exist"
Sep 15 16:47:10.664065 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:10.664046    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-v42zh"]
Sep 15 16:47:12.159032 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:47:12.159000    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" path="/var/lib/kubelet/pods/cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254/volumes"
Sep 15 16:49:15.438975 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.438894    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"]
Sep 15 16:49:15.439411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.439254    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kube-rbac-proxy"
Sep 15 16:49:15.439411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.439268    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kube-rbac-proxy"
Sep 15 16:49:15.439411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.439281    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="storage-initializer"
Sep 15 16:49:15.439411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.439289    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="storage-initializer"
Sep 15 16:49:15.439411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.439300    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container"
Sep 15 16:49:15.439411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.439305    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container"
Sep 15 16:49:15.439411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.439354    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kube-rbac-proxy"
Sep 15 16:49:15.439411 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.439364    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc6b4c17-e67c-42a2-a2c1-4e4ef9ffb254" containerName="kserve-container"
Sep 15 16:49:15.442389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.442368    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.444801 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.444773    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:49:15.444907 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.444772    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Sep 15 16:49:15.444958 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.444934    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Sep 15 16:49:15.445120 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.445101    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:49:15.445803 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.445787    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:49:15.445894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.445801    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:49:15.445894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.445816    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:49:15.451894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.451871    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"]
Sep 15 16:49:15.551615 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.551559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.551757 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.551640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.551757 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.551668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-api-access-q6ksr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.551757 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.551730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.551890 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.551766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f45f9-fb29-4afd-ad45-8b1628973b2c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.652461 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.652434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.652621 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.652466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-api-access-q6ksr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.652621 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.652492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.652621 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.652527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f45f9-fb29-4afd-ad45-8b1628973b2c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.652621 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:49:15.652618    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-serving-cert: secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 15 16:49:15.652804 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.652627    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.652804 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:49:15.652677    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls podName:310f45f9-fb29-4afd-ad45-8b1628973b2c nodeName:}" failed. No retries permitted until 2026-09-15 16:49:16.152657913 +0000 UTC m=+1344.520483099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls") pod "isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" (UID: "310f45f9-fb29-4afd-ad45-8b1628973b2c") : secret "isvc-autogluon-v2-noslash-predictor-serving-cert" not found
Sep 15 16:49:15.653038 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.653016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f45f9-fb29-4afd-ad45-8b1628973b2c-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.653213 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.653195    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.653248 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.653198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:15.663381 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:15.663352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-api-access-q6ksr\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:16.155923 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:16.155888    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:16.158605 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:16.158581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:16.353147 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:16.353112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:49:16.625868 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:16.625841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"]
Sep 15 16:49:16.627659 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:49:16.627635    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod310f45f9_fb29_4afd_ad45_8b1628973b2c.slice/crio-9b98c75c84fa3db6495e562ab8e842441a6bc95c54d5c901749d1944a5a5a419 WatchSource:0}: Error finding container 9b98c75c84fa3db6495e562ab8e842441a6bc95c54d5c901749d1944a5a5a419: Status 404 returned error can't find the container with id 9b98c75c84fa3db6495e562ab8e842441a6bc95c54d5c901749d1944a5a5a419
Sep 15 16:49:17.195619 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:17.195559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" event={"ID":"310f45f9-fb29-4afd-ad45-8b1628973b2c","Type":"ContainerStarted","Data":"ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21"}
Sep 15 16:49:17.195812 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:17.195645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" event={"ID":"310f45f9-fb29-4afd-ad45-8b1628973b2c","Type":"ContainerStarted","Data":"9b98c75c84fa3db6495e562ab8e842441a6bc95c54d5c901749d1944a5a5a419"}
Sep 15 16:49:46.325764 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:46.325671    2566 generic.go:356] "Generic (PLEG): container finished" podID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerID="ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21" exitCode=0
Sep 15 16:49:46.325764 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:46.325727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" event={"ID":"310f45f9-fb29-4afd-ad45-8b1628973b2c","Type":"ContainerDied","Data":"ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21"}
Sep 15 16:49:46.326938 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:49:46.326919    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:51:39.905121 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:51:39.905081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" event={"ID":"310f45f9-fb29-4afd-ad45-8b1628973b2c","Type":"ContainerStarted","Data":"373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3"}
Sep 15 16:51:39.905590 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:51:39.905145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" event={"ID":"310f45f9-fb29-4afd-ad45-8b1628973b2c","Type":"ContainerStarted","Data":"7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27"}
Sep 15 16:51:39.905590 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:51:39.905211    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:51:39.925504 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:51:39.925454    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" podStartSLOduration=3.02564238 podStartE2EDuration="2m24.925440207s" totalImagesPullingTime="1m52.585844056s" totalInitContainerRuntime="29.313953771s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:49:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:51:39.922990622 +0000 UTC m=+1488.290815822" watchObservedRunningTime="2026-09-15 16:51:39.925440207 +0000 UTC m=+1488.293265477"
Sep 15 16:51:40.909776 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:51:40.909746    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:51:46.919955 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:51:46.919924    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:51:52.146131 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:51:52.146103    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:51:52.152334 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:51:52.152309    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:53:17.157276 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:17.157243    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:53:28.194723 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:28.194682    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.39:8080/v2/models/isvc-autogluon-v2-noslash/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 15 16:53:31.553478 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:31.553437    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"]
Sep 15 16:53:31.553898 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:31.553827    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kserve-container" containerID="cri-o://7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27" gracePeriod=30
Sep 15 16:53:31.553975 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:31.553856    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kube-rbac-proxy" containerID="cri-o://373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3" gracePeriod=30
Sep 15 16:53:31.915642 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:31.915598    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused"
Sep 15 16:53:32.405735 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.405695    2566 generic.go:356] "Generic (PLEG): container finished" podID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerID="373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3" exitCode=2
Sep 15 16:53:32.405897 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.405756    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" event={"ID":"310f45f9-fb29-4afd-ad45-8b1628973b2c","Type":"ContainerDied","Data":"373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3"}
Sep 15 16:53:32.902335 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.902310    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:53:32.979201 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.979132    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-api-access-q6ksr\") pod \"310f45f9-fb29-4afd-ad45-8b1628973b2c\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") "
Sep 15 16:53:32.979201 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.979167    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-cabundle-cert\") pod \"310f45f9-fb29-4afd-ad45-8b1628973b2c\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") "
Sep 15 16:53:32.979201 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.979186    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-rbac-proxy-sar-config\") pod \"310f45f9-fb29-4afd-ad45-8b1628973b2c\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") "
Sep 15 16:53:32.979456 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.979229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls\") pod \"310f45f9-fb29-4afd-ad45-8b1628973b2c\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") "
Sep 15 16:53:32.979456 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.979257    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f45f9-fb29-4afd-ad45-8b1628973b2c-kserve-provision-location\") pod \"310f45f9-fb29-4afd-ad45-8b1628973b2c\" (UID: \"310f45f9-fb29-4afd-ad45-8b1628973b2c\") "
Sep 15 16:53:32.979627 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.979603    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "310f45f9-fb29-4afd-ad45-8b1628973b2c" (UID: "310f45f9-fb29-4afd-ad45-8b1628973b2c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:53:32.979694 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.979621    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "310f45f9-fb29-4afd-ad45-8b1628973b2c" (UID: "310f45f9-fb29-4afd-ad45-8b1628973b2c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:53:32.981166 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.981144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-api-access-q6ksr" (OuterVolumeSpecName: "kube-api-access-q6ksr") pod "310f45f9-fb29-4afd-ad45-8b1628973b2c" (UID: "310f45f9-fb29-4afd-ad45-8b1628973b2c"). InnerVolumeSpecName "kube-api-access-q6ksr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:53:32.981336 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.981317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "310f45f9-fb29-4afd-ad45-8b1628973b2c" (UID: "310f45f9-fb29-4afd-ad45-8b1628973b2c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:53:32.983801 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:32.983779    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/310f45f9-fb29-4afd-ad45-8b1628973b2c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "310f45f9-fb29-4afd-ad45-8b1628973b2c" (UID: "310f45f9-fb29-4afd-ad45-8b1628973b2c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:53:33.080109 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.080067    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/310f45f9-fb29-4afd-ad45-8b1628973b2c-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:53:33.080109 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.080107    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/310f45f9-fb29-4afd-ad45-8b1628973b2c-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:53:33.080109 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.080116    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ksr\" (UniqueName: \"kubernetes.io/projected/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-api-access-q6ksr\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:53:33.080302 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.080126    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:53:33.080302 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.080135    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/310f45f9-fb29-4afd-ad45-8b1628973b2c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:53:33.412354 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.412325    2566 generic.go:356] "Generic (PLEG): container finished" podID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerID="7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27" exitCode=0
Sep 15 16:53:33.412534 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.412386    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"
Sep 15 16:53:33.412534 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.412408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" event={"ID":"310f45f9-fb29-4afd-ad45-8b1628973b2c","Type":"ContainerDied","Data":"7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27"}
Sep 15 16:53:33.412534 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.412478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h" event={"ID":"310f45f9-fb29-4afd-ad45-8b1628973b2c","Type":"ContainerDied","Data":"9b98c75c84fa3db6495e562ab8e842441a6bc95c54d5c901749d1944a5a5a419"}
Sep 15 16:53:33.412534 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.412517    2566 scope.go:117] "RemoveContainer" containerID="373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3"
Sep 15 16:53:33.421767 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.421749    2566 scope.go:117] "RemoveContainer" containerID="7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27"
Sep 15 16:53:33.429505 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.429484    2566 scope.go:117] "RemoveContainer" containerID="ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21"
Sep 15 16:53:33.434902 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.434882    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"]
Sep 15 16:53:33.437918 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.437743    2566 scope.go:117] "RemoveContainer" containerID="373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3"
Sep 15 16:53:33.438095 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:53:33.438079    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3\": container with ID starting with 373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3 not found: ID does not exist" containerID="373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3"
Sep 15 16:53:33.438144 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.438104    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3"} err="failed to get container status \"373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3\": rpc error: code = NotFound desc = could not find container \"373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3\": container with ID starting with 373784b1f440ba46d98ec9449abb64582344dc23f5dc6835825c7ea2485a32c3 not found: ID does not exist"
Sep 15 16:53:33.438144 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.438120    2566 scope.go:117] "RemoveContainer" containerID="7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27"
Sep 15 16:53:33.438415 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:53:33.438400    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27\": container with ID starting with 7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27 not found: ID does not exist" containerID="7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27"
Sep 15 16:53:33.438481 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.438418    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27"} err="failed to get container status \"7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27\": rpc error: code = NotFound desc = could not find container \"7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27\": container with ID starting with 7ba069335362a9e343f1699f1cf74e7c7f235435421bf3aa3c533c83add4ed27 not found: ID does not exist"
Sep 15 16:53:33.438481 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.438430    2566 scope.go:117] "RemoveContainer" containerID="ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21"
Sep 15 16:53:33.438761 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:53:33.438733    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21\": container with ID starting with ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21 not found: ID does not exist" containerID="ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21"
Sep 15 16:53:33.438832 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.438762    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21"} err="failed to get container status \"ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21\": rpc error: code = NotFound desc = could not find container \"ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21\": container with ID starting with ef7587120e4e724e497dfa09b256a346d98106b5e3b58d2fe9a4b6e09b838f21 not found: ID does not exist"
Sep 15 16:53:33.439650 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:33.439632    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-5499c79f9f-g5c4h"]
Sep 15 16:53:34.158768 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:34.158727    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" path="/var/lib/kubelet/pods/310f45f9-fb29-4afd-ad45-8b1628973b2c/volumes"
Sep 15 16:53:36.706171 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706133    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"]
Sep 15 16:53:36.706647 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706609    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kube-rbac-proxy"
Sep 15 16:53:36.706647 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706628    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kube-rbac-proxy"
Sep 15 16:53:36.706647 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706644    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="storage-initializer"
Sep 15 16:53:36.706818 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706653    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="storage-initializer"
Sep 15 16:53:36.706818 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706684    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kserve-container"
Sep 15 16:53:36.706818 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706696    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kserve-container"
Sep 15 16:53:36.706818 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706788    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kube-rbac-proxy"
Sep 15 16:53:36.706818 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.706802    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="310f45f9-fb29-4afd-ad45-8b1628973b2c" containerName="kserve-container"
Sep 15 16:53:36.710447 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.710425    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.713090 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.713064    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Sep 15 16:53:36.713203 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.713119    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:53:36.713203 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.713158    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:53:36.713203 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.713186    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:53:36.713377 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.713067    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Sep 15 16:53:36.713377 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.713237    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:53:36.714027 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.714008    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:53:36.718562 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.718538    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"]
Sep 15 16:53:36.809343 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.809316    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.809470 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.809347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.809470 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.809368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed56182a-47e4-4743-805b-6f8b60578c9b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.809551 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.809484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7m5\" (UniqueName: \"kubernetes.io/projected/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-api-access-kq7m5\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.809551 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.809510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.910350 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.910324    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed56182a-47e4-4743-805b-6f8b60578c9b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.910467 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.910398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7m5\" (UniqueName: \"kubernetes.io/projected/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-api-access-kq7m5\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.910467 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.910419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.910467 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.910445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.910661 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.910470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.910661 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:53:36.910592    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 15 16:53:36.910661 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:53:36.910657    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls podName:ed56182a-47e4-4743-805b-6f8b60578c9b nodeName:}" failed. No retries permitted until 2026-09-15 16:53:37.410638216 +0000 UTC m=+1605.778463394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" (UID: "ed56182a-47e4-4743-805b-6f8b60578c9b") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Sep 15 16:53:36.910885 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.910853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed56182a-47e4-4743-805b-6f8b60578c9b-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.911107 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.911088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.911141 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.911103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:36.920119 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:36.920103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7m5\" (UniqueName: \"kubernetes.io/projected/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-api-access-kq7m5\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:37.415178 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:37.415142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:37.417514 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:37.417494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:37.621721 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:37.621686    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:37.893429 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:37.893408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"]
Sep 15 16:53:37.895170 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:53:37.895144    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded56182a_47e4_4743_805b_6f8b60578c9b.slice/crio-49358c7c2b63b8ee9c8ed4256a1aaa44725c6540b62d8b2efcc44e2597201872 WatchSource:0}: Error finding container 49358c7c2b63b8ee9c8ed4256a1aaa44725c6540b62d8b2efcc44e2597201872: Status 404 returned error can't find the container with id 49358c7c2b63b8ee9c8ed4256a1aaa44725c6540b62d8b2efcc44e2597201872
Sep 15 16:53:38.436600 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:38.436549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" event={"ID":"ed56182a-47e4-4743-805b-6f8b60578c9b","Type":"ContainerStarted","Data":"2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20"}
Sep 15 16:53:38.436758 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:38.436643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" event={"ID":"ed56182a-47e4-4743-805b-6f8b60578c9b","Type":"ContainerStarted","Data":"49358c7c2b63b8ee9c8ed4256a1aaa44725c6540b62d8b2efcc44e2597201872"}
Sep 15 16:53:48.485888 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:48.485844    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerID="2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20" exitCode=0
Sep 15 16:53:48.486282 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:48.485915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" event={"ID":"ed56182a-47e4-4743-805b-6f8b60578c9b","Type":"ContainerDied","Data":"2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20"}
Sep 15 16:53:49.492535 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:49.492496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" event={"ID":"ed56182a-47e4-4743-805b-6f8b60578c9b","Type":"ContainerStarted","Data":"0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6"}
Sep 15 16:53:49.492894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:49.492581    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" event={"ID":"ed56182a-47e4-4743-805b-6f8b60578c9b","Type":"ContainerStarted","Data":"35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0"}
Sep 15 16:53:49.492894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:49.492791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:49.511908 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:49.511854    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" podStartSLOduration=3.821946576 podStartE2EDuration="13.511839016s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.68989244s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:53:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:53:49.509728058 +0000 UTC m=+1617.877553270" watchObservedRunningTime="2026-09-15 16:53:49.511839016 +0000 UTC m=+1617.879664214"
Sep 15 16:53:50.497909 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:50.497876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:50.499213 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:50.499179    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 15 16:53:51.503838 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:51.503798    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 15 16:53:56.508787 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:56.508757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:53:56.509264 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:53:56.509233    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Sep 15 16:54:06.511133 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:06.511104    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:54:28.815630 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:28.815593    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"]
Sep 15 16:54:28.816092 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:28.815889    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kserve-container" containerID="cri-o://35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0" gracePeriod=30
Sep 15 16:54:28.816092 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:28.815963    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kube-rbac-proxy" containerID="cri-o://0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6" gracePeriod=30
Sep 15 16:54:29.681740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:29.681700    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerID="0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6" exitCode=2
Sep 15 16:54:29.681928 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:29.681771    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" event={"ID":"ed56182a-47e4-4743-805b-6f8b60578c9b","Type":"ContainerDied","Data":"0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6"}
Sep 15 16:54:30.468319 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.468297    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:54:30.546601 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.546503    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kq7m5\" (UniqueName: \"kubernetes.io/projected/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-api-access-kq7m5\") pod \"ed56182a-47e4-4743-805b-6f8b60578c9b\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") "
Sep 15 16:54:30.546601 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.546549    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed56182a-47e4-4743-805b-6f8b60578c9b-kserve-provision-location\") pod \"ed56182a-47e4-4743-805b-6f8b60578c9b\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") "
Sep 15 16:54:30.546817 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.546617    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-cabundle-cert\") pod \"ed56182a-47e4-4743-805b-6f8b60578c9b\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") "
Sep 15 16:54:30.546817 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.546639    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls\") pod \"ed56182a-47e4-4743-805b-6f8b60578c9b\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") "
Sep 15 16:54:30.546817 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.546681    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-rbac-proxy-sar-config\") pod \"ed56182a-47e4-4743-805b-6f8b60578c9b\" (UID: \"ed56182a-47e4-4743-805b-6f8b60578c9b\") "
Sep 15 16:54:30.547075 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.547050    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ed56182a-47e4-4743-805b-6f8b60578c9b" (UID: "ed56182a-47e4-4743-805b-6f8b60578c9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:54:30.547151 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.547091    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed56182a-47e4-4743-805b-6f8b60578c9b" (UID: "ed56182a-47e4-4743-805b-6f8b60578c9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:54:30.547662 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.547635    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed56182a-47e4-4743-805b-6f8b60578c9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ed56182a-47e4-4743-805b-6f8b60578c9b" (UID: "ed56182a-47e4-4743-805b-6f8b60578c9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:54:30.548758 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.548735    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-api-access-kq7m5" (OuterVolumeSpecName: "kube-api-access-kq7m5") pod "ed56182a-47e4-4743-805b-6f8b60578c9b" (UID: "ed56182a-47e4-4743-805b-6f8b60578c9b"). InnerVolumeSpecName "kube-api-access-kq7m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:54:30.548837 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.548759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed56182a-47e4-4743-805b-6f8b60578c9b" (UID: "ed56182a-47e4-4743-805b-6f8b60578c9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:54:30.647657 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.647630    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:54:30.647657 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.647654    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed56182a-47e4-4743-805b-6f8b60578c9b-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:54:30.647800 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.647664    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:54:30.647800 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.647673    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq7m5\" (UniqueName: \"kubernetes.io/projected/ed56182a-47e4-4743-805b-6f8b60578c9b-kube-api-access-kq7m5\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:54:30.647800 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.647682    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ed56182a-47e4-4743-805b-6f8b60578c9b-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:54:30.688276 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.688246    2566 generic.go:356] "Generic (PLEG): container finished" podID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerID="35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0" exitCode=0
Sep 15 16:54:30.688389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.688291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" event={"ID":"ed56182a-47e4-4743-805b-6f8b60578c9b","Type":"ContainerDied","Data":"35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0"}
Sep 15 16:54:30.688389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.688305    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"
Sep 15 16:54:30.688389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.688355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z" event={"ID":"ed56182a-47e4-4743-805b-6f8b60578c9b","Type":"ContainerDied","Data":"49358c7c2b63b8ee9c8ed4256a1aaa44725c6540b62d8b2efcc44e2597201872"}
Sep 15 16:54:30.688389 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.688385    2566 scope.go:117] "RemoveContainer" containerID="0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6"
Sep 15 16:54:30.696935 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.696919    2566 scope.go:117] "RemoveContainer" containerID="35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0"
Sep 15 16:54:30.704380 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.704363    2566 scope.go:117] "RemoveContainer" containerID="2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20"
Sep 15 16:54:30.709063 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.709042    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"]
Sep 15 16:54:30.712243 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.712216    2566 scope.go:117] "RemoveContainer" containerID="0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6"
Sep 15 16:54:30.712507 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:54:30.712488    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6\": container with ID starting with 0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6 not found: ID does not exist" containerID="0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6"
Sep 15 16:54:30.712603 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.712518    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6"} err="failed to get container status \"0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6\": rpc error: code = NotFound desc = could not find container \"0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6\": container with ID starting with 0cfb1fe09eea2059820dccb3f36e347108825169442ff62fd4fa36ba3be9cae6 not found: ID does not exist"
Sep 15 16:54:30.712603 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.712544    2566 scope.go:117] "RemoveContainer" containerID="35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0"
Sep 15 16:54:30.712844 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:54:30.712828    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0\": container with ID starting with 35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0 not found: ID does not exist" containerID="35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0"
Sep 15 16:54:30.712912 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.712852    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0"} err="failed to get container status \"35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0\": rpc error: code = NotFound desc = could not find container \"35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0\": container with ID starting with 35a5e3c31a6586b82376201fef9a82685958b1146a5be38232f256f6f7cc34b0 not found: ID does not exist"
Sep 15 16:54:30.712912 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.712871    2566 scope.go:117] "RemoveContainer" containerID="2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20"
Sep 15 16:54:30.713103 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:54:30.713085    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20\": container with ID starting with 2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20 not found: ID does not exist" containerID="2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20"
Sep 15 16:54:30.713142 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.713116    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20"} err="failed to get container status \"2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20\": rpc error: code = NotFound desc = could not find container \"2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20\": container with ID starting with 2f8dd6c5e568ca4ca01d890fb44b515d675ebda9f966098ff51176ecb3430d20 not found: ID does not exist"
Sep 15 16:54:30.713400 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:30.713383    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-6cf79d9f7d-9jr7z"]
Sep 15 16:54:32.158177 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:32.158143    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" path="/var/lib/kubelet/pods/ed56182a-47e4-4743-805b-6f8b60578c9b/volumes"
Sep 15 16:54:34.018620 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.018591    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"]
Sep 15 16:54:34.018988 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.018927    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kserve-container"
Sep 15 16:54:34.018988 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.018939    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kserve-container"
Sep 15 16:54:34.018988 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.018946    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kube-rbac-proxy"
Sep 15 16:54:34.018988 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.018954    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kube-rbac-proxy"
Sep 15 16:54:34.018988 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.018969    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="storage-initializer"
Sep 15 16:54:34.018988 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.018975    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="storage-initializer"
Sep 15 16:54:34.019182 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.019036    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kserve-container"
Sep 15 16:54:34.019182 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.019046    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed56182a-47e4-4743-805b-6f8b60578c9b" containerName="kube-rbac-proxy"
Sep 15 16:54:34.022316 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.022300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.024906 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.024881    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:54:34.025031 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.024917    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:54:34.026225 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.026200    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Sep 15 16:54:34.027219 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.026545    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:54:34.027219 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.026595    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:54:34.027219 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.026935    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:54:34.027219 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.027061    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Sep 15 16:54:34.036515 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.036488    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"]
Sep 15 16:54:34.077105 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.077079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.077212 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.077124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.077212 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.077158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.077212 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.077190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.077212 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.077208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcr9m\" (UniqueName: \"kubernetes.io/projected/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-api-access-tcr9m\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.178230 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.178199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.178381 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.178241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.178381 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.178266    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.178381 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.178302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.178381 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.178327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcr9m\" (UniqueName: \"kubernetes.io/projected/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-api-access-tcr9m\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.178676 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.178651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.178963 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.178943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.179033 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.179013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.180927 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.180898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.186031 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.186010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcr9m\" (UniqueName: \"kubernetes.io/projected/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-api-access-tcr9m\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.337884 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.337816    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:34.610532 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.610504    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"]
Sep 15 16:54:34.612987 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:54:34.612954    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4e5d5e_8d76_4e19_a8c1_772e3171359a.slice/crio-9953b1fb0c64da56447ae92c714862c6a83a02cd900dd12820c6c3b0f1cecf34 WatchSource:0}: Error finding container 9953b1fb0c64da56447ae92c714862c6a83a02cd900dd12820c6c3b0f1cecf34: Status 404 returned error can't find the container with id 9953b1fb0c64da56447ae92c714862c6a83a02cd900dd12820c6c3b0f1cecf34
Sep 15 16:54:34.707740 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.707704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" event={"ID":"3b4e5d5e-8d76-4e19-a8c1-772e3171359a","Type":"ContainerStarted","Data":"cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e"}
Sep 15 16:54:34.707864 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:34.707781    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" event={"ID":"3b4e5d5e-8d76-4e19-a8c1-772e3171359a","Type":"ContainerStarted","Data":"9953b1fb0c64da56447ae92c714862c6a83a02cd900dd12820c6c3b0f1cecf34"}
Sep 15 16:54:39.729905 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:39.729872    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerID="cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e" exitCode=0
Sep 15 16:54:39.730440 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:39.729944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" event={"ID":"3b4e5d5e-8d76-4e19-a8c1-772e3171359a","Type":"ContainerDied","Data":"cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e"}
Sep 15 16:54:40.743094 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:40.743064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" event={"ID":"3b4e5d5e-8d76-4e19-a8c1-772e3171359a","Type":"ContainerStarted","Data":"26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e"}
Sep 15 16:54:40.743492 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:40.743144    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" event={"ID":"3b4e5d5e-8d76-4e19-a8c1-772e3171359a","Type":"ContainerStarted","Data":"20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a"}
Sep 15 16:54:40.743492 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:40.743412    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:40.764110 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:40.764064    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" podStartSLOduration=2.129668447 podStartE2EDuration="6.764052369s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.634383922s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:54:40.762630037 +0000 UTC m=+1669.130455236" watchObservedRunningTime="2026-09-15 16:54:40.764052369 +0000 UTC m=+1669.131877568"
Sep 15 16:54:41.747977 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:41.747943    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:41.749155 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:41.749131    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 15 16:54:42.752264 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:42.752218    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 15 16:54:47.757449 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:47.757418    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:54:47.757971 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:47.757938    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Sep 15 16:54:57.758969 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:54:57.758937    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:55:09.043802 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:09.043768    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"]
Sep 15 16:55:09.044259 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:09.044061    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kserve-container" containerID="cri-o://20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a" gracePeriod=30
Sep 15 16:55:09.044259 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:09.044115    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kube-rbac-proxy" containerID="cri-o://26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e" gracePeriod=30
Sep 15 16:55:09.877614 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:09.877559    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerID="26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e" exitCode=2
Sep 15 16:55:09.877818 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:09.877633    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" event={"ID":"3b4e5d5e-8d76-4e19-a8c1-772e3171359a","Type":"ContainerDied","Data":"26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e"}
Sep 15 16:55:10.386520 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.386490    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:55:10.466082 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.466016    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-rbac-proxy-sar-config\") pod \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") "
Sep 15 16:55:10.466082 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.466054    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcr9m\" (UniqueName: \"kubernetes.io/projected/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-api-access-tcr9m\") pod \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") "
Sep 15 16:55:10.466272 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.466113    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kserve-provision-location\") pod \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") "
Sep 15 16:55:10.466272 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.466129    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-proxy-tls\") pod \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") "
Sep 15 16:55:10.466272 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.466151    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-cabundle-cert\") pod \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\" (UID: \"3b4e5d5e-8d76-4e19-a8c1-772e3171359a\") "
Sep 15 16:55:10.466548 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.466523    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b4e5d5e-8d76-4e19-a8c1-772e3171359a" (UID: "3b4e5d5e-8d76-4e19-a8c1-772e3171359a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:55:10.466678 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.466653    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3b4e5d5e-8d76-4e19-a8c1-772e3171359a" (UID: "3b4e5d5e-8d76-4e19-a8c1-772e3171359a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:55:10.467182 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.467156    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3b4e5d5e-8d76-4e19-a8c1-772e3171359a" (UID: "3b4e5d5e-8d76-4e19-a8c1-772e3171359a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:55:10.468183 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.468162    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b4e5d5e-8d76-4e19-a8c1-772e3171359a" (UID: "3b4e5d5e-8d76-4e19-a8c1-772e3171359a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:55:10.468269 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.468253    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-api-access-tcr9m" (OuterVolumeSpecName: "kube-api-access-tcr9m") pod "3b4e5d5e-8d76-4e19-a8c1-772e3171359a" (UID: "3b4e5d5e-8d76-4e19-a8c1-772e3171359a"). InnerVolumeSpecName "kube-api-access-tcr9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:55:10.567223 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.567200    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:55:10.567223 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.567221    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcr9m\" (UniqueName: \"kubernetes.io/projected/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kube-api-access-tcr9m\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:55:10.567349 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.567230    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:55:10.567349 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.567240    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:55:10.567349 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.567250    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3b4e5d5e-8d76-4e19-a8c1-772e3171359a-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:55:10.883488 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.883455    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerID="20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a" exitCode=0
Sep 15 16:55:10.883684 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.883509    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"
Sep 15 16:55:10.883684 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.883545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" event={"ID":"3b4e5d5e-8d76-4e19-a8c1-772e3171359a","Type":"ContainerDied","Data":"20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a"}
Sep 15 16:55:10.883684 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.883645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw" event={"ID":"3b4e5d5e-8d76-4e19-a8c1-772e3171359a","Type":"ContainerDied","Data":"9953b1fb0c64da56447ae92c714862c6a83a02cd900dd12820c6c3b0f1cecf34"}
Sep 15 16:55:10.883855 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.883685    2566 scope.go:117] "RemoveContainer" containerID="26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e"
Sep 15 16:55:10.892071 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.892057    2566 scope.go:117] "RemoveContainer" containerID="20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a"
Sep 15 16:55:10.899547 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.899531    2566 scope.go:117] "RemoveContainer" containerID="cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e"
Sep 15 16:55:10.905330 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.905310    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"]
Sep 15 16:55:10.907530 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.907509    2566 scope.go:117] "RemoveContainer" containerID="26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e"
Sep 15 16:55:10.907837 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:55:10.907813    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e\": container with ID starting with 26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e not found: ID does not exist" containerID="26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e"
Sep 15 16:55:10.907930 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.907843    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e"} err="failed to get container status \"26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e\": rpc error: code = NotFound desc = could not find container \"26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e\": container with ID starting with 26d95fe64747a0615efd88475d138c02ab16831026dbbfce0c0f0ebf662c880e not found: ID does not exist"
Sep 15 16:55:10.907930 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.907861    2566 scope.go:117] "RemoveContainer" containerID="20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a"
Sep 15 16:55:10.908092 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:55:10.908076    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a\": container with ID starting with 20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a not found: ID does not exist" containerID="20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a"
Sep 15 16:55:10.908180 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.908093    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a"} err="failed to get container status \"20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a\": rpc error: code = NotFound desc = could not find container \"20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a\": container with ID starting with 20b0a1c639ac30e28cb2e22441810c61b171fc7cc71f7913a7f8d64ed6a6479a not found: ID does not exist"
Sep 15 16:55:10.908180 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.908106    2566 scope.go:117] "RemoveContainer" containerID="cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e"
Sep 15 16:55:10.908359 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:55:10.908340    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e\": container with ID starting with cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e not found: ID does not exist" containerID="cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e"
Sep 15 16:55:10.908416 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.908363    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e"} err="failed to get container status \"cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e\": rpc error: code = NotFound desc = could not find container \"cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e\": container with ID starting with cc1ba981ba0b1f447e8bd2d28bd294abe91cb5f3f728a03ac4247453cda1196e not found: ID does not exist"
Sep 15 16:55:10.909560 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:10.909539    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-cfmdw"]
Sep 15 16:55:12.158715 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:12.158684    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" path="/var/lib/kubelet/pods/3b4e5d5e-8d76-4e19-a8c1-772e3171359a/volumes"
Sep 15 16:55:14.303446 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303371    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"]
Sep 15 16:55:14.303805 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303723    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kserve-container"
Sep 15 16:55:14.303805 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303736    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kserve-container"
Sep 15 16:55:14.303805 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303751    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kube-rbac-proxy"
Sep 15 16:55:14.303805 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303757    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kube-rbac-proxy"
Sep 15 16:55:14.303805 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303765    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="storage-initializer"
Sep 15 16:55:14.303805 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303773    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="storage-initializer"
Sep 15 16:55:14.304002 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303826    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kube-rbac-proxy"
Sep 15 16:55:14.304002 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.303838    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b4e5d5e-8d76-4e19-a8c1-772e3171359a" containerName="kserve-container"
Sep 15 16:55:14.308686 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.308667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.311156 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.311130    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:55:14.311156 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.311146    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 15 16:55:14.311391 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.311376    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:55:14.311754 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.311737    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 15 16:55:14.311754 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.311750    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:55:14.311881 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.311765    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:55:14.311881 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.311771    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:55:14.317587 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.317533    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"]
Sep 15 16:55:14.393771 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.393733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.393771 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.393777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df38b415-a393-4daa-8acf-7294155718f6-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.393995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.393840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.393995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.393913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.393995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.393953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fxqt\" (UniqueName: \"kubernetes.io/projected/df38b415-a393-4daa-8acf-7294155718f6-kube-api-access-9fxqt\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.494310 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.494279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.494463 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.494316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df38b415-a393-4daa-8acf-7294155718f6-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.494463 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.494354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.494463 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.494378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.494463 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.494402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fxqt\" (UniqueName: \"kubernetes.io/projected/df38b415-a393-4daa-8acf-7294155718f6-kube-api-access-9fxqt\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.494699 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:55:14.494521    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-create-predictor-serving-cert: secret "isvc-canary-create-predictor-serving-cert" not found
Sep 15 16:55:14.494699 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:55:14.494596    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls podName:df38b415-a393-4daa-8acf-7294155718f6 nodeName:}" failed. No retries permitted until 2026-09-15 16:55:14.994556601 +0000 UTC m=+1703.362381792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls") pod "isvc-canary-create-predictor-7d5b67cc88-pzlbm" (UID: "df38b415-a393-4daa-8acf-7294155718f6") : secret "isvc-canary-create-predictor-serving-cert" not found
Sep 15 16:55:14.494773 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.494750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df38b415-a393-4daa-8acf-7294155718f6-kserve-provision-location\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.495063 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.495042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-cabundle-cert\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.495103 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.495052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.502995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.502965    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fxqt\" (UniqueName: \"kubernetes.io/projected/df38b415-a393-4daa-8acf-7294155718f6-kube-api-access-9fxqt\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:14.999084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:14.999045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:15.001471 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:15.001432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls\") pod \"isvc-canary-create-predictor-7d5b67cc88-pzlbm\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:15.224759 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:15.224710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:15.501308 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:15.501216    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"]
Sep 15 16:55:15.503762 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:55:15.503725    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf38b415_a393_4daa_8acf_7294155718f6.slice/crio-3d5cb1af60680c67618fa547498ffa35216dd68aa7a0f77c29ab8abc59c4aee4 WatchSource:0}: Error finding container 3d5cb1af60680c67618fa547498ffa35216dd68aa7a0f77c29ab8abc59c4aee4: Status 404 returned error can't find the container with id 3d5cb1af60680c67618fa547498ffa35216dd68aa7a0f77c29ab8abc59c4aee4
Sep 15 16:55:15.908996 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:15.908954    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" event={"ID":"df38b415-a393-4daa-8acf-7294155718f6","Type":"ContainerStarted","Data":"d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18"}
Sep 15 16:55:15.909189 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:15.909018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" event={"ID":"df38b415-a393-4daa-8acf-7294155718f6","Type":"ContainerStarted","Data":"3d5cb1af60680c67618fa547498ffa35216dd68aa7a0f77c29ab8abc59c4aee4"}
Sep 15 16:55:19.929609 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:19.929548    2566 generic.go:356] "Generic (PLEG): container finished" podID="df38b415-a393-4daa-8acf-7294155718f6" containerID="d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18" exitCode=0
Sep 15 16:55:19.930096 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:19.929629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" event={"ID":"df38b415-a393-4daa-8acf-7294155718f6","Type":"ContainerDied","Data":"d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18"}
Sep 15 16:55:19.930921 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:19.930906    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 16:55:35.019509 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:35.019475    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" event={"ID":"df38b415-a393-4daa-8acf-7294155718f6","Type":"ContainerStarted","Data":"a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2"}
Sep 15 16:55:35.019895 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:35.019538    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" event={"ID":"df38b415-a393-4daa-8acf-7294155718f6","Type":"ContainerStarted","Data":"ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e"}
Sep 15 16:55:35.019895 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:35.019825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:35.019895 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:35.019881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:35.021393 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:35.021366    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:55:35.041158 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:35.041112    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podStartSLOduration=3.241096185 podStartE2EDuration="21.041095579s" totalImagesPullingTime="14.084109148s" totalInitContainerRuntime="3.715890246s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:55:35.039252343 +0000 UTC m=+1723.407077758" watchObservedRunningTime="2026-09-15 16:55:35.041095579 +0000 UTC m=+1723.408920778"
Sep 15 16:55:36.024817 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:36.024782    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:55:41.030821 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:41.030789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:55:41.031282 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:41.031260    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:55:51.032040 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:55:51.031996    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:56:01.032184 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:01.032145    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:56:11.032176 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:11.032134    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:56:21.032215 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:21.032171    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:56:31.031858 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:31.031816    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:56:41.031795 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:41.031755    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused"
Sep 15 16:56:51.032747 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:51.032717    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:56:52.179466 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:52.179434    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:56:52.186402 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:52.186379    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 16:56:54.310460 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:54.310425    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"]
Sep 15 16:56:54.310862 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:54.310733    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container" containerID="cri-o://ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e" gracePeriod=30
Sep 15 16:56:54.310862 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:54.310770    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kube-rbac-proxy" containerID="cri-o://a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2" gracePeriod=30
Sep 15 16:56:55.389750 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:55.389711    2566 generic.go:356] "Generic (PLEG): container finished" podID="df38b415-a393-4daa-8acf-7294155718f6" containerID="a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2" exitCode=2
Sep 15 16:56:55.390129 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:55.389779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" event={"ID":"df38b415-a393-4daa-8acf-7294155718f6","Type":"ContainerDied","Data":"a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2"}
Sep 15 16:56:56.025553 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:56.025508    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused"
Sep 15 16:56:58.651184 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.651160    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:56:58.737199 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.737122    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-kube-rbac-proxy-sar-config\") pod \"df38b415-a393-4daa-8acf-7294155718f6\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") "
Sep 15 16:56:58.737199 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.737170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df38b415-a393-4daa-8acf-7294155718f6-kserve-provision-location\") pod \"df38b415-a393-4daa-8acf-7294155718f6\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") "
Sep 15 16:56:58.737407 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.737231    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fxqt\" (UniqueName: \"kubernetes.io/projected/df38b415-a393-4daa-8acf-7294155718f6-kube-api-access-9fxqt\") pod \"df38b415-a393-4daa-8acf-7294155718f6\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") "
Sep 15 16:56:58.737407 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.737268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls\") pod \"df38b415-a393-4daa-8acf-7294155718f6\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") "
Sep 15 16:56:58.737407 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.737311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-cabundle-cert\") pod \"df38b415-a393-4daa-8acf-7294155718f6\" (UID: \"df38b415-a393-4daa-8acf-7294155718f6\") "
Sep 15 16:56:58.737558 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.737529    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df38b415-a393-4daa-8acf-7294155718f6" (UID: "df38b415-a393-4daa-8acf-7294155718f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:56:58.737674 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.737533    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df38b415-a393-4daa-8acf-7294155718f6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df38b415-a393-4daa-8acf-7294155718f6" (UID: "df38b415-a393-4daa-8acf-7294155718f6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:56:58.737749 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.737731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "df38b415-a393-4daa-8acf-7294155718f6" (UID: "df38b415-a393-4daa-8acf-7294155718f6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:56:58.739371 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.739349    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df38b415-a393-4daa-8acf-7294155718f6-kube-api-access-9fxqt" (OuterVolumeSpecName: "kube-api-access-9fxqt") pod "df38b415-a393-4daa-8acf-7294155718f6" (UID: "df38b415-a393-4daa-8acf-7294155718f6"). InnerVolumeSpecName "kube-api-access-9fxqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:56:58.739456 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.739433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df38b415-a393-4daa-8acf-7294155718f6" (UID: "df38b415-a393-4daa-8acf-7294155718f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:56:58.838865 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.838836    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fxqt\" (UniqueName: \"kubernetes.io/projected/df38b415-a393-4daa-8acf-7294155718f6-kube-api-access-9fxqt\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:56:58.838865 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.838860    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df38b415-a393-4daa-8acf-7294155718f6-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:56:58.838865 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.838870    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:56:58.839101 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.838879    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df38b415-a393-4daa-8acf-7294155718f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:56:58.839101 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:58.838886    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df38b415-a393-4daa-8acf-7294155718f6-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:56:59.409985 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.409945    2566 generic.go:356] "Generic (PLEG): container finished" podID="df38b415-a393-4daa-8acf-7294155718f6" containerID="ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e" exitCode=0
Sep 15 16:56:59.409985 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.409976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" event={"ID":"df38b415-a393-4daa-8acf-7294155718f6","Type":"ContainerDied","Data":"ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e"}
Sep 15 16:56:59.410199 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.410002    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"
Sep 15 16:56:59.410199 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.410043    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm" event={"ID":"df38b415-a393-4daa-8acf-7294155718f6","Type":"ContainerDied","Data":"3d5cb1af60680c67618fa547498ffa35216dd68aa7a0f77c29ab8abc59c4aee4"}
Sep 15 16:56:59.410199 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.410078    2566 scope.go:117] "RemoveContainer" containerID="a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2"
Sep 15 16:56:59.421118 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.421017    2566 scope.go:117] "RemoveContainer" containerID="ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e"
Sep 15 16:56:59.429950 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.429927    2566 scope.go:117] "RemoveContainer" containerID="d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18"
Sep 15 16:56:59.438364 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.438335    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"]
Sep 15 16:56:59.439749 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.439730    2566 scope.go:117] "RemoveContainer" containerID="a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2"
Sep 15 16:56:59.440075 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:56:59.440047    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2\": container with ID starting with a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2 not found: ID does not exist" containerID="a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2"
Sep 15 16:56:59.440250 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.440083    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2"} err="failed to get container status \"a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2\": rpc error: code = NotFound desc = could not find container \"a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2\": container with ID starting with a1ad9e6c06fb237540840f495e2091cbceabf129bda096894ed709c3cd01e9a2 not found: ID does not exist"
Sep 15 16:56:59.440250 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.440101    2566 scope.go:117] "RemoveContainer" containerID="ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e"
Sep 15 16:56:59.441293 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:56:59.440530    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e\": container with ID starting with ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e not found: ID does not exist" containerID="ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e"
Sep 15 16:56:59.441293 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.440672    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e"} err="failed to get container status \"ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e\": rpc error: code = NotFound desc = could not find container \"ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e\": container with ID starting with ea91c9335189811ed383b8f51d4fc17ebf1bceced10ac4dcb24b06b4ce932a8e not found: ID does not exist"
Sep 15 16:56:59.441293 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.440698    2566 scope.go:117] "RemoveContainer" containerID="d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18"
Sep 15 16:56:59.441293 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:56:59.441223    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18\": container with ID starting with d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18 not found: ID does not exist" containerID="d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18"
Sep 15 16:56:59.441293 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.441258    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18"} err="failed to get container status \"d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18\": rpc error: code = NotFound desc = could not find container \"d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18\": container with ID starting with d3bb34806d7dfe0cb7ce4006b659364c06727177b8a0e1368a50d93c82acea18 not found: ID does not exist"
Sep 15 16:56:59.443342 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.443321    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7d5b67cc88-pzlbm"]
Sep 15 16:56:59.499845 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.499818    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"]
Sep 15 16:56:59.500147 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.500122    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kube-rbac-proxy"
Sep 15 16:56:59.500147 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.500138    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kube-rbac-proxy"
Sep 15 16:56:59.500147 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.500152    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container"
Sep 15 16:56:59.500321 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.500158    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container"
Sep 15 16:56:59.500321 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.500167    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="storage-initializer"
Sep 15 16:56:59.500321 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.500174    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="storage-initializer"
Sep 15 16:56:59.500321 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.500225    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kserve-container"
Sep 15 16:56:59.500321 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.500234    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="df38b415-a393-4daa-8acf-7294155718f6" containerName="kube-rbac-proxy"
Sep 15 16:56:59.503244 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.503225    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.505492 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.505475    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 15 16:56:59.505631 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.505606    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 16:56:59.505790 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.505769    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 15 16:56:59.505900 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.505883    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 16:56:59.505951 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.505883    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 16:56:59.505995 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.505965    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 16:56:59.506039 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.505892    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 16:56:59.513910 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.513888    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"]
Sep 15 16:56:59.544705 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.544681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.544837 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.544710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.544837 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.544769    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.544966 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.544855    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw5s7\" (UniqueName: \"kubernetes.io/projected/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-api-access-sw5s7\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.544966 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.544905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.646225 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.646179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.646225 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.646219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.646417 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.646264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.646417 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.646283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw5s7\" (UniqueName: \"kubernetes.io/projected/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-api-access-sw5s7\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.646417 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.646304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.646828 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.646795    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kserve-provision-location\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.647072 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.647055    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-cabundle-cert\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.647114 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.647073    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.648583 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.648548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-proxy-tls\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.653970 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.653947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw5s7\" (UniqueName: \"kubernetes.io/projected/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-api-access-sw5s7\") pod \"isvc-canary-promote-predictor-6d6bc676fb-t2j5c\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:56:59.814261 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:56:59.814175    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:57:00.086352 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:00.086267    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"]
Sep 15 16:57:00.089556 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:57:00.089529    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd614ee53_bc5a_4ca2_aae9_2fcfa7c09c74.slice/crio-035559aa7f703411c1bd74509bfda177be9e14df5cedc3300a7c65ccb2da254b WatchSource:0}: Error finding container 035559aa7f703411c1bd74509bfda177be9e14df5cedc3300a7c65ccb2da254b: Status 404 returned error can't find the container with id 035559aa7f703411c1bd74509bfda177be9e14df5cedc3300a7c65ccb2da254b
Sep 15 16:57:00.159715 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:00.159686    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df38b415-a393-4daa-8acf-7294155718f6" path="/var/lib/kubelet/pods/df38b415-a393-4daa-8acf-7294155718f6/volumes"
Sep 15 16:57:00.416435 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:00.416395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" event={"ID":"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74","Type":"ContainerStarted","Data":"db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4"}
Sep 15 16:57:00.416644 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:00.416472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" event={"ID":"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74","Type":"ContainerStarted","Data":"035559aa7f703411c1bd74509bfda177be9e14df5cedc3300a7c65ccb2da254b"}
Sep 15 16:57:04.436504 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:04.436468    2566 generic.go:356] "Generic (PLEG): container finished" podID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerID="db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4" exitCode=0
Sep 15 16:57:04.436905 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:04.436546    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" event={"ID":"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74","Type":"ContainerDied","Data":"db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4"}
Sep 15 16:57:05.442851 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:05.442815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" event={"ID":"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74","Type":"ContainerStarted","Data":"eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc"}
Sep 15 16:57:05.443264 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:05.442880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" event={"ID":"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74","Type":"ContainerStarted","Data":"773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0"}
Sep 15 16:57:05.443264 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:05.443082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:57:05.462483 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:05.462349    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podStartSLOduration=2.554100982 podStartE2EDuration="6.46233113s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.908230148s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:56:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:57:05.462077934 +0000 UTC m=+1813.829903134" watchObservedRunningTime="2026-09-15 16:57:05.46233113 +0000 UTC m=+1813.830156331"
Sep 15 16:57:06.447984 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:06.447952    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:57:06.449474 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:06.449445    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:57:07.452717 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:07.452673    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:57:12.457964 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:12.457925    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:57:12.458529 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:12.458501    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:57:18.501079 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.501044    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"]
Sep 15 16:57:18.503743 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.503721    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.507087 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.507065    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Sep 15 16:57:18.507286 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.507273    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Sep 15 16:57:18.514502 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.514472    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"]
Sep 15 16:57:18.600386 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.600350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h754n\" (UniqueName: \"kubernetes.io/projected/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-api-access-h754n\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.600535 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.600457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.600535 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.600482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.600535 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.600510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a707cd-ac09-4ff3-836a-0482b36937b7-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.600673 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.600548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.701733 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.701698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.701733 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.701735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.701910 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.701753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a707cd-ac09-4ff3-836a-0482b36937b7-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.701910 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.701804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.701910 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.701850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h754n\" (UniqueName: \"kubernetes.io/projected/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-api-access-h754n\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.701910 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:57:18.701866    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 15 16:57:18.702131 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:57:18.701934    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls podName:20a707cd-ac09-4ff3-836a-0482b36937b7 nodeName:}" failed. No retries permitted until 2026-09-15 16:57:19.201915478 +0000 UTC m=+1827.569740654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-d4cmm" (UID: "20a707cd-ac09-4ff3-836a-0482b36937b7") : secret "isvc-tensorflow-predictor-serving-cert" not found
Sep 15 16:57:18.702211 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.702191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a707cd-ac09-4ff3-836a-0482b36937b7-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.702405 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.702385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.702524 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.702507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:18.710069 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:18.710045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h754n\" (UniqueName: \"kubernetes.io/projected/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-api-access-h754n\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:19.205333 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:19.205296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:19.207769 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:19.207750    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-d4cmm\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:19.414720 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:19.414690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:19.685524 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:19.685494    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"]
Sep 15 16:57:19.687354 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:57:19.687324    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a707cd_ac09_4ff3_836a_0482b36937b7.slice/crio-3484a197247a514df91f6bae9b35e81357d3f9702b0b97f80c6313e6c3ad6bd5 WatchSource:0}: Error finding container 3484a197247a514df91f6bae9b35e81357d3f9702b0b97f80c6313e6c3ad6bd5: Status 404 returned error can't find the container with id 3484a197247a514df91f6bae9b35e81357d3f9702b0b97f80c6313e6c3ad6bd5
Sep 15 16:57:20.514557 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:20.514514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" event={"ID":"20a707cd-ac09-4ff3-836a-0482b36937b7","Type":"ContainerStarted","Data":"7332ff6d7bc1b40b2379a2f615cf66a09c34d43cd177eb555c3fa7053f4d2b40"}
Sep 15 16:57:20.514771 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:20.514618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" event={"ID":"20a707cd-ac09-4ff3-836a-0482b36937b7","Type":"ContainerStarted","Data":"3484a197247a514df91f6bae9b35e81357d3f9702b0b97f80c6313e6c3ad6bd5"}
Sep 15 16:57:22.458898 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:22.458860    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:57:24.534437 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:24.534342    2566 generic.go:356] "Generic (PLEG): container finished" podID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerID="7332ff6d7bc1b40b2379a2f615cf66a09c34d43cd177eb555c3fa7053f4d2b40" exitCode=0
Sep 15 16:57:24.534437 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:24.534419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" event={"ID":"20a707cd-ac09-4ff3-836a-0482b36937b7","Type":"ContainerDied","Data":"7332ff6d7bc1b40b2379a2f615cf66a09c34d43cd177eb555c3fa7053f4d2b40"}
Sep 15 16:57:28.558158 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:28.558079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" event={"ID":"20a707cd-ac09-4ff3-836a-0482b36937b7","Type":"ContainerStarted","Data":"e39de021f34a1fe410ed43bb1c78b5768623c2a1e9677682c31d3d1a69594908"}
Sep 15 16:57:28.558158 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:28.558139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" event={"ID":"20a707cd-ac09-4ff3-836a-0482b36937b7","Type":"ContainerStarted","Data":"59db26ed82637ccc0b47f9bb3e9fc3f762c46dee0d19fbecc92e7a8600a64324"}
Sep 15 16:57:28.558589 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:28.558414    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:28.558654 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:28.558604    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:28.560228 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:28.560206    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 15 16:57:28.581649 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:28.581607    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podStartSLOduration=2.400683134 podStartE2EDuration="10.581593672s" totalImagesPullingTime="3.691329956s" totalInitContainerRuntime="4.489580582s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:57:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:57:28.580791676 +0000 UTC m=+1836.948616879" watchObservedRunningTime="2026-09-15 16:57:28.581593672 +0000 UTC m=+1836.949418925"
Sep 15 16:57:29.563504 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:29.563445    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 15 16:57:32.459253 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:32.459213    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:57:34.567925 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:34.567895    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:34.568405 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:34.568377    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.44:8080: connect: connection refused"
Sep 15 16:57:42.459156 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:42.459109    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:57:44.568940 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:44.568910    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:57:52.459026 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:52.458983    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:57:59.464310 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:59.464269    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"]
Sep 15 16:57:59.464792 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:59.464591    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kserve-container" containerID="cri-o://59db26ed82637ccc0b47f9bb3e9fc3f762c46dee0d19fbecc92e7a8600a64324" gracePeriod=30
Sep 15 16:57:59.465424 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:59.465029    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy" containerID="cri-o://e39de021f34a1fe410ed43bb1c78b5768623c2a1e9677682c31d3d1a69594908" gracePeriod=30
Sep 15 16:57:59.564360 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:59.564315    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 15 16:57:59.705984 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:59.705954    2566 generic.go:356] "Generic (PLEG): container finished" podID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerID="e39de021f34a1fe410ed43bb1c78b5768623c2a1e9677682c31d3d1a69594908" exitCode=2
Sep 15 16:57:59.706084 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:57:59.706002    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" event={"ID":"20a707cd-ac09-4ff3-836a-0482b36937b7","Type":"ContainerDied","Data":"e39de021f34a1fe410ed43bb1c78b5768623c2a1e9677682c31d3d1a69594908"}
Sep 15 16:58:02.458933 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:02.458895    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:58:04.563920 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:04.563872    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 15 16:58:09.564800 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:09.564747    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 15 16:58:09.565181 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:09.564938    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:58:12.459070 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:12.459029    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.43:8080: connect: connection refused"
Sep 15 16:58:14.564105 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:14.564020    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 15 16:58:19.155222 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:19.155192    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:58:19.563881 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:19.563794    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 15 16:58:24.563873 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:24.563826    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 15 16:58:29.507679 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.507649    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"]
Sep 15 16:58:29.508016 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.507920    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container" containerID="cri-o://773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0" gracePeriod=30
Sep 15 16:58:29.508086 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.507984    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kube-rbac-proxy" containerID="cri-o://eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc" gracePeriod=30
Sep 15 16:58:29.564467 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.564436    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.44:8643/healthz\": dial tcp 10.132.0.44:8643: connect: connection refused"
Sep 15 16:58:29.600907 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.600868    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"]
Sep 15 16:58:29.603765 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.603744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.606108 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.606085    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 15 16:58:29.614345 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.614320    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"]
Sep 15 16:58:29.677908 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.677878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.678023 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.677914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.678023 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.677946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22x8x\" (UniqueName: \"kubernetes.io/projected/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-api-access-22x8x\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.678023 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.677998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.678023 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.678016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.779010 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.778938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.779010 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.778968    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.779010 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.778998    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22x8x\" (UniqueName: \"kubernetes.io/projected/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-api-access-22x8x\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.779266 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.779020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.779266 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.779051    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.779377 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.779322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.779747 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.779722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.779837 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.779790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.781494 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.781472    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.787629 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.787598    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22x8x\" (UniqueName: \"kubernetes.io/projected/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-api-access-22x8x\") pod \"isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:29.842838 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.842815    2566 generic.go:356] "Generic (PLEG): container finished" podID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerID="eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc" exitCode=2
Sep 15 16:58:29.842929 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.842887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" event={"ID":"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74","Type":"ContainerDied","Data":"eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc"}
Sep 15 16:58:29.845193 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.845170    2566 generic.go:356] "Generic (PLEG): container finished" podID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerID="59db26ed82637ccc0b47f9bb3e9fc3f762c46dee0d19fbecc92e7a8600a64324" exitCode=137
Sep 15 16:58:29.845271 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.845238    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" event={"ID":"20a707cd-ac09-4ff3-836a-0482b36937b7","Type":"ContainerDied","Data":"59db26ed82637ccc0b47f9bb3e9fc3f762c46dee0d19fbecc92e7a8600a64324"}
Sep 15 16:58:29.924634 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:29.924610    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:30.111352 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.111322    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:58:30.184739 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.184709    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-rbac-proxy-sar-config\") pod \"20a707cd-ac09-4ff3-836a-0482b36937b7\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") "
Sep 15 16:58:30.184991 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.184764    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-cabundle-cert\") pod \"20a707cd-ac09-4ff3-836a-0482b36937b7\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") "
Sep 15 16:58:30.184991 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.184812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h754n\" (UniqueName: \"kubernetes.io/projected/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-api-access-h754n\") pod \"20a707cd-ac09-4ff3-836a-0482b36937b7\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") "
Sep 15 16:58:30.184991 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.184838    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a707cd-ac09-4ff3-836a-0482b36937b7-kserve-provision-location\") pod \"20a707cd-ac09-4ff3-836a-0482b36937b7\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") "
Sep 15 16:58:30.184991 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.184898    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls\") pod \"20a707cd-ac09-4ff3-836a-0482b36937b7\" (UID: \"20a707cd-ac09-4ff3-836a-0482b36937b7\") "
Sep 15 16:58:30.185272 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.185131    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "20a707cd-ac09-4ff3-836a-0482b36937b7" (UID: "20a707cd-ac09-4ff3-836a-0482b36937b7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:58:30.185272 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.185136    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "20a707cd-ac09-4ff3-836a-0482b36937b7" (UID: "20a707cd-ac09-4ff3-836a-0482b36937b7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:58:30.187028 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.186999    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-api-access-h754n" (OuterVolumeSpecName: "kube-api-access-h754n") pod "20a707cd-ac09-4ff3-836a-0482b36937b7" (UID: "20a707cd-ac09-4ff3-836a-0482b36937b7"). InnerVolumeSpecName "kube-api-access-h754n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:58:30.187126 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.187060    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "20a707cd-ac09-4ff3-836a-0482b36937b7" (UID: "20a707cd-ac09-4ff3-836a-0482b36937b7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:58:30.195640 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.195622    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"]
Sep 15 16:58:30.196796 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.196765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a707cd-ac09-4ff3-836a-0482b36937b7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "20a707cd-ac09-4ff3-836a-0482b36937b7" (UID: "20a707cd-ac09-4ff3-836a-0482b36937b7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:58:30.197895 ip-10-0-132-85 kubenswrapper[2566]: W0915 16:58:30.197872    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5a1736a_5e16_4ddd_8f27_87ba94e7f023.slice/crio-b91faf0c05752ef89118b20734e5034a52a277418d4c0f5764d8780248d33e17 WatchSource:0}: Error finding container b91faf0c05752ef89118b20734e5034a52a277418d4c0f5764d8780248d33e17: Status 404 returned error can't find the container with id b91faf0c05752ef89118b20734e5034a52a277418d4c0f5764d8780248d33e17
Sep 15 16:58:30.286487 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.286445    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:30.286487 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.286475    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/20a707cd-ac09-4ff3-836a-0482b36937b7-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:30.286767 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.286519    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h754n\" (UniqueName: \"kubernetes.io/projected/20a707cd-ac09-4ff3-836a-0482b36937b7-kube-api-access-h754n\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:30.286767 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.286554    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/20a707cd-ac09-4ff3-836a-0482b36937b7-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:30.286767 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.286589    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/20a707cd-ac09-4ff3-836a-0482b36937b7-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:30.853005 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.852920    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"
Sep 15 16:58:30.853453 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.852915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm" event={"ID":"20a707cd-ac09-4ff3-836a-0482b36937b7","Type":"ContainerDied","Data":"3484a197247a514df91f6bae9b35e81357d3f9702b0b97f80c6313e6c3ad6bd5"}
Sep 15 16:58:30.853453 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.853095    2566 scope.go:117] "RemoveContainer" containerID="e39de021f34a1fe410ed43bb1c78b5768623c2a1e9677682c31d3d1a69594908"
Sep 15 16:58:30.854931 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.854892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" event={"ID":"a5a1736a-5e16-4ddd-8f27-87ba94e7f023","Type":"ContainerStarted","Data":"596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a"}
Sep 15 16:58:30.855062 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.854966    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" event={"ID":"a5a1736a-5e16-4ddd-8f27-87ba94e7f023","Type":"ContainerStarted","Data":"b91faf0c05752ef89118b20734e5034a52a277418d4c0f5764d8780248d33e17"}
Sep 15 16:58:30.863801 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.863783    2566 scope.go:117] "RemoveContainer" containerID="59db26ed82637ccc0b47f9bb3e9fc3f762c46dee0d19fbecc92e7a8600a64324"
Sep 15 16:58:30.871446 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.871424    2566 scope.go:117] "RemoveContainer" containerID="7332ff6d7bc1b40b2379a2f615cf66a09c34d43cd177eb555c3fa7053f4d2b40"
Sep 15 16:58:30.889812 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.889790    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"]
Sep 15 16:58:30.891947 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:30.891925    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-d4cmm"]
Sep 15 16:58:32.158781 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:32.158743    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" path="/var/lib/kubelet/pods/20a707cd-ac09-4ff3-836a-0482b36937b7/volumes"
Sep 15 16:58:32.453104 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:32.453011    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.43:8643/healthz\": dial tcp 10.132.0.43:8643: connect: connection refused"
Sep 15 16:58:34.055025 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.055003    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:58:34.217677 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.217647    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kserve-provision-location\") pod \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") "
Sep 15 16:58:34.217859 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.217713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-rbac-proxy-sar-config\") pod \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") "
Sep 15 16:58:34.217859 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.217741    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sw5s7\" (UniqueName: \"kubernetes.io/projected/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-api-access-sw5s7\") pod \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") "
Sep 15 16:58:34.217859 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.217788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-proxy-tls\") pod \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") "
Sep 15 16:58:34.217859 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.217825    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-cabundle-cert\") pod \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\" (UID: \"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74\") "
Sep 15 16:58:34.218081 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.217962    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" (UID: "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 16:58:34.218081 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.218057    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" (UID: "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:58:34.218267 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.218240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" (UID: "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 16:58:34.219933 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.219914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-api-access-sw5s7" (OuterVolumeSpecName: "kube-api-access-sw5s7") pod "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" (UID: "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74"). InnerVolumeSpecName "kube-api-access-sw5s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 16:58:34.220031 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.219915    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" (UID: "d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 16:58:34.318580 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.318543    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:34.318687 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.318566    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:34.318687 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.318599    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:34.318687 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.318612    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:34.318687 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.318627    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw5s7\" (UniqueName: \"kubernetes.io/projected/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74-kube-api-access-sw5s7\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 16:58:34.878131 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.878101    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerID="596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a" exitCode=0
Sep 15 16:58:34.878262 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.878173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" event={"ID":"a5a1736a-5e16-4ddd-8f27-87ba94e7f023","Type":"ContainerDied","Data":"596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a"}
Sep 15 16:58:34.880246 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.880226    2566 generic.go:356] "Generic (PLEG): container finished" podID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerID="773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0" exitCode=0
Sep 15 16:58:34.880358 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.880294    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"
Sep 15 16:58:34.880358 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.880304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" event={"ID":"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74","Type":"ContainerDied","Data":"773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0"}
Sep 15 16:58:34.880488 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.880373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c" event={"ID":"d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74","Type":"ContainerDied","Data":"035559aa7f703411c1bd74509bfda177be9e14df5cedc3300a7c65ccb2da254b"}
Sep 15 16:58:34.880488 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.880412    2566 scope.go:117] "RemoveContainer" containerID="eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc"
Sep 15 16:58:34.891599 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.891584    2566 scope.go:117] "RemoveContainer" containerID="773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0"
Sep 15 16:58:34.905273 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.905257    2566 scope.go:117] "RemoveContainer" containerID="db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4"
Sep 15 16:58:34.910513 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.910481    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"]
Sep 15 16:58:34.914616 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.914564    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-6d6bc676fb-t2j5c"]
Sep 15 16:58:34.917892 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.917876    2566 scope.go:117] "RemoveContainer" containerID="eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc"
Sep 15 16:58:34.918169 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:58:34.918147    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc\": container with ID starting with eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc not found: ID does not exist" containerID="eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc"
Sep 15 16:58:34.918255 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.918180    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc"} err="failed to get container status \"eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc\": rpc error: code = NotFound desc = could not find container \"eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc\": container with ID starting with eff71bd7377b733767987e34ae9f50c1dcf051c907ffa3050e3e34f81fe22afc not found: ID does not exist"
Sep 15 16:58:34.918255 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.918205    2566 scope.go:117] "RemoveContainer" containerID="773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0"
Sep 15 16:58:34.918520 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:58:34.918500    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0\": container with ID starting with 773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0 not found: ID does not exist" containerID="773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0"
Sep 15 16:58:34.918588 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.918526    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0"} err="failed to get container status \"773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0\": rpc error: code = NotFound desc = could not find container \"773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0\": container with ID starting with 773c2d337bbb1e4ee659e33a5fbc8883ffbc1093ce944da3c1038b02e5b42ad0 not found: ID does not exist"
Sep 15 16:58:34.918588 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.918542    2566 scope.go:117] "RemoveContainer" containerID="db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4"
Sep 15 16:58:34.918838 ip-10-0-132-85 kubenswrapper[2566]: E0915 16:58:34.918821    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4\": container with ID starting with db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4 not found: ID does not exist" containerID="db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4"
Sep 15 16:58:34.918894 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:34.918844    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4"} err="failed to get container status \"db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4\": rpc error: code = NotFound desc = could not find container \"db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4\": container with ID starting with db43b4904a91c624cf03a586c1455b3e91ef9e247d03cc8c1a72180bcff48ec4 not found: ID does not exist"
Sep 15 16:58:35.888368 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:35.888335    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" event={"ID":"a5a1736a-5e16-4ddd-8f27-87ba94e7f023","Type":"ContainerStarted","Data":"e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db"}
Sep 15 16:58:35.888778 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:35.888397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" event={"ID":"a5a1736a-5e16-4ddd-8f27-87ba94e7f023","Type":"ContainerStarted","Data":"d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5"}
Sep 15 16:58:35.888778 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:35.888683    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:35.888885 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:35.888827    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:35.890300 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:35.890277    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:58:35.906403 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:35.906357    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podStartSLOduration=2.379515544 podStartE2EDuration="6.90634319s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.526827646s" isStatefulPod=false podCreationTimestamp="2026-09-15 16:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 16:58:35.905454554 +0000 UTC m=+1904.273279765" watchObservedRunningTime="2026-09-15 16:58:35.90634319 +0000 UTC m=+1904.274168388"
Sep 15 16:58:36.159163 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:36.159083    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" path="/var/lib/kubelet/pods/d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74/volumes"
Sep 15 16:58:36.892635 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:36.892594    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:58:41.896720 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:41.896685    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:58:41.897331 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:41.897302    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:58:51.898181 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:58:51.898139    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:59:01.897317 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:01.897278    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:59:11.897190 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:11.897148    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:59:21.897776 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:21.897732    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:59:31.897756 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:31.897716    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:59:41.897239 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:41.897193    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:59:46.154208 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:46.154117    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.45:8080: connect: connection refused"
Sep 15 16:59:56.158691 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:56.158663    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 16:59:59.720563 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:59.720529    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"]
Sep 15 16:59:59.721152 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:59.720887    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container" containerID="cri-o://d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5" gracePeriod=30
Sep 15 16:59:59.721152 ip-10-0-132-85 kubenswrapper[2566]: I0915 16:59:59.721000    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kube-rbac-proxy" containerID="cri-o://e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db" gracePeriod=30
Sep 15 17:00:00.268476 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:00.268442    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerID="e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db" exitCode=2
Sep 15 17:00:00.268665 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:00.268519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" event={"ID":"a5a1736a-5e16-4ddd-8f27-87ba94e7f023","Type":"ContainerDied","Data":"e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db"}
Sep 15 17:00:01.893004 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:01.892966    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.45:8643/healthz\": dial tcp 10.132.0.45:8643: connect: connection refused"
Sep 15 17:00:04.167220 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.167191    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 17:00:04.226553 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.226478    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kserve-provision-location\") pod \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") "
Sep 15 17:00:04.226553 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.226532    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-proxy-tls\") pod \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") "
Sep 15 17:00:04.226823 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.226614    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22x8x\" (UniqueName: \"kubernetes.io/projected/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-api-access-22x8x\") pod \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") "
Sep 15 17:00:04.226823 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.226640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-rbac-proxy-sar-config\") pod \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") "
Sep 15 17:00:04.226823 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.226678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-cabundle-cert\") pod \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\" (UID: \"a5a1736a-5e16-4ddd-8f27-87ba94e7f023\") "
Sep 15 17:00:04.226982 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.226929    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5a1736a-5e16-4ddd-8f27-87ba94e7f023" (UID: "a5a1736a-5e16-4ddd-8f27-87ba94e7f023"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:00:04.227068 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.227044    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5a1736a-5e16-4ddd-8f27-87ba94e7f023" (UID: "a5a1736a-5e16-4ddd-8f27-87ba94e7f023"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:00:04.227106 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.227039    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5a1736a-5e16-4ddd-8f27-87ba94e7f023" (UID: "a5a1736a-5e16-4ddd-8f27-87ba94e7f023"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:00:04.228646 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.228621    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5a1736a-5e16-4ddd-8f27-87ba94e7f023" (UID: "a5a1736a-5e16-4ddd-8f27-87ba94e7f023"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:00:04.228738 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.228708    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-api-access-22x8x" (OuterVolumeSpecName: "kube-api-access-22x8x") pod "a5a1736a-5e16-4ddd-8f27-87ba94e7f023" (UID: "a5a1736a-5e16-4ddd-8f27-87ba94e7f023"). InnerVolumeSpecName "kube-api-access-22x8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:00:04.287768 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.287744    2566 generic.go:356] "Generic (PLEG): container finished" podID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerID="d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5" exitCode=0
Sep 15 17:00:04.287890 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.287799    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"
Sep 15 17:00:04.287890 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.287835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" event={"ID":"a5a1736a-5e16-4ddd-8f27-87ba94e7f023","Type":"ContainerDied","Data":"d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5"}
Sep 15 17:00:04.288012 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.287908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg" event={"ID":"a5a1736a-5e16-4ddd-8f27-87ba94e7f023","Type":"ContainerDied","Data":"b91faf0c05752ef89118b20734e5034a52a277418d4c0f5764d8780248d33e17"}
Sep 15 17:00:04.288012 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.287950    2566 scope.go:117] "RemoveContainer" containerID="e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db"
Sep 15 17:00:04.297165 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.297148    2566 scope.go:117] "RemoveContainer" containerID="d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5"
Sep 15 17:00:04.304631 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.304609    2566 scope.go:117] "RemoveContainer" containerID="596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a"
Sep 15 17:00:04.309151 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.309131    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"]
Sep 15 17:00:04.312268 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.312245    2566 scope.go:117] "RemoveContainer" containerID="e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db"
Sep 15 17:00:04.312549 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:00:04.312527    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db\": container with ID starting with e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db not found: ID does not exist" containerID="e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db"
Sep 15 17:00:04.312744 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.312559    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db"} err="failed to get container status \"e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db\": rpc error: code = NotFound desc = could not find container \"e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db\": container with ID starting with e829d486008d02ed66313406d7c601e63ab84c48c9aa9a9f4bf1ad7c545695db not found: ID does not exist"
Sep 15 17:00:04.312744 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.312607    2566 scope.go:117] "RemoveContainer" containerID="d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5"
Sep 15 17:00:04.312824 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.312805    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-d567b8dc5-8ffxg"]
Sep 15 17:00:04.312916 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:00:04.312897    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5\": container with ID starting with d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5 not found: ID does not exist" containerID="d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5"
Sep 15 17:00:04.312955 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.312922    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5"} err="failed to get container status \"d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5\": rpc error: code = NotFound desc = could not find container \"d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5\": container with ID starting with d8db2bc5b3dd594e50641fa911a34cd7391b41a3abcc4d74637e15fb3df239f5 not found: ID does not exist"
Sep 15 17:00:04.312955 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.312937    2566 scope.go:117] "RemoveContainer" containerID="596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a"
Sep 15 17:00:04.313166 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:00:04.313149    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a\": container with ID starting with 596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a not found: ID does not exist" containerID="596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a"
Sep 15 17:00:04.313216 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.313170    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a"} err="failed to get container status \"596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a\": rpc error: code = NotFound desc = could not find container \"596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a\": container with ID starting with 596aae124394d5af3931a1f5b0ca44805e49523ff0b8bfa8bbe0f7e17571789a not found: ID does not exist"
Sep 15 17:00:04.327687 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.327661    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22x8x\" (UniqueName: \"kubernetes.io/projected/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-api-access-22x8x\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:00:04.327687 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.327690    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:00:04.327789 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.327699    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:00:04.327789 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.327707    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:00:04.327789 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.327716    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5a1736a-5e16-4ddd-8f27-87ba94e7f023-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:00:04.905869 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.905837    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"]
Sep 15 17:00:04.906167 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906155    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906168    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906177    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="storage-initializer"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906183    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="storage-initializer"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906193    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906199    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906205    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906211    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906216    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906221    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906228    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="storage-initializer"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="storage-initializer"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906241    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906246    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906254    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kserve-container"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906259    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kserve-container"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906270    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="storage-initializer"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906275    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="storage-initializer"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906325    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906333    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kserve-container"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906341    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d614ee53-bc5a-4ca2-aae9-2fcfa7c09c74" containerName="kserve-container"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906348    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906355    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="20a707cd-ac09-4ff3-836a-0482b36937b7" containerName="kube-rbac-proxy"
Sep 15 17:00:04.906533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.906359    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" containerName="kserve-container"
Sep 15 17:00:04.910946 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.910926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:04.913619 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.913557    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:00:04.913739 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.913628    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:00:04.913739 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.913641    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 15 17:00:04.913844 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.913586    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:00:04.913885 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.913866    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:00:04.914036 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.914015    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 15 17:00:04.914530 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.914511    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:00:04.919332 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.919312    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"]
Sep 15 17:00:04.932022 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.931998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:04.932110 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.932025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:04.932110 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.932053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:04.932110 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.932079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:04.932244 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.932148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rz2hr\" (UniqueName: \"kubernetes.io/projected/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-api-access-rz2hr\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:04.986405 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.986371    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"]
Sep 15 17:00:04.990065 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.990045    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:04.992230 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:04.992213    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 15 17:00:05.002307 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.002285    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"]
Sep 15 17:00:05.032680 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.032658    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.032800 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.032690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe76ae1-caba-4adc-8998-cac717fcabe0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.032800 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.032730    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rz2hr\" (UniqueName: \"kubernetes.io/projected/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-api-access-rz2hr\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.032800 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.032779    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wp8h\" (UniqueName: \"kubernetes.io/projected/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-api-access-2wp8h\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.032908 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.032840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe76ae1-caba-4adc-8998-cac717fcabe0-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.032908 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.032862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.032908 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.032903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.033023 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.032933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.033023 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:00:05.033017    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 15 17:00:05.033105 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.033052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.033105 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:00:05.033072    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls podName:5ddc00d3-f8dc-4f9f-883c-a221f9062c18 nodeName:}" failed. No retries permitted until 2026-09-15 17:00:05.533054535 +0000 UTC m=+1993.900879713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls") pod "isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" (UID: "5ddc00d3-f8dc-4f9f-883c-a221f9062c18") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Sep 15 17:00:05.033210 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.033107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.033284 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.033267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.033383 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.033367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.033530 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.033511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.041419 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.041391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rz2hr\" (UniqueName: \"kubernetes.io/projected/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-api-access-rz2hr\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.133887 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.133850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe76ae1-caba-4adc-8998-cac717fcabe0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.134050 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.133905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wp8h\" (UniqueName: \"kubernetes.io/projected/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-api-access-2wp8h\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.134050 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.133955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe76ae1-caba-4adc-8998-cac717fcabe0-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.134050 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.133974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.134221 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.134048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.134338 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.134318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe76ae1-caba-4adc-8998-cac717fcabe0-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.134720 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.134696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-cabundle-cert\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.134811 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.134722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.136534 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.136513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe76ae1-caba-4adc-8998-cac717fcabe0-proxy-tls\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.141764 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.141741    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wp8h\" (UniqueName: \"kubernetes.io/projected/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-api-access-2wp8h\") pod \"isvc-canary-rollback-predictor-6b76f97c96-fxx9m\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.299713 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.299628    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:05.537223 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.537187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.539535 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.539504    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:05.577649 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.577624    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"]
Sep 15 17:00:05.579322 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:00:05.579293    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe76ae1_caba_4adc_8998_cac717fcabe0.slice/crio-642579b72bbf09dd068b57493e85b9cd4f07da6a75734bcbc541fde8cbfb649b WatchSource:0}: Error finding container 642579b72bbf09dd068b57493e85b9cd4f07da6a75734bcbc541fde8cbfb649b: Status 404 returned error can't find the container with id 642579b72bbf09dd068b57493e85b9cd4f07da6a75734bcbc541fde8cbfb649b
Sep 15 17:00:05.822350 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:05.822311    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:06.095450 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:06.095419    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"]
Sep 15 17:00:06.097018 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:00:06.096990    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ddc00d3_f8dc_4f9f_883c_a221f9062c18.slice/crio-26b34b6b383641246a604987ec4aa6e906892eb8dcd1e4f56d9c145900f070d4 WatchSource:0}: Error finding container 26b34b6b383641246a604987ec4aa6e906892eb8dcd1e4f56d9c145900f070d4: Status 404 returned error can't find the container with id 26b34b6b383641246a604987ec4aa6e906892eb8dcd1e4f56d9c145900f070d4
Sep 15 17:00:06.159536 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:06.159505    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5a1736a-5e16-4ddd-8f27-87ba94e7f023" path="/var/lib/kubelet/pods/a5a1736a-5e16-4ddd-8f27-87ba94e7f023/volumes"
Sep 15 17:00:06.301627 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:06.301585    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" event={"ID":"5ddc00d3-f8dc-4f9f-883c-a221f9062c18","Type":"ContainerStarted","Data":"efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c"}
Sep 15 17:00:06.302094 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:06.301666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" event={"ID":"5ddc00d3-f8dc-4f9f-883c-a221f9062c18","Type":"ContainerStarted","Data":"26b34b6b383641246a604987ec4aa6e906892eb8dcd1e4f56d9c145900f070d4"}
Sep 15 17:00:06.303403 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:06.303377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" event={"ID":"6fe76ae1-caba-4adc-8998-cac717fcabe0","Type":"ContainerStarted","Data":"a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d"}
Sep 15 17:00:06.303548 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:06.303437    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" event={"ID":"6fe76ae1-caba-4adc-8998-cac717fcabe0","Type":"ContainerStarted","Data":"642579b72bbf09dd068b57493e85b9cd4f07da6a75734bcbc541fde8cbfb649b"}
Sep 15 17:00:10.322421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:10.322386    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerID="efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c" exitCode=0
Sep 15 17:00:10.322880 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:10.322464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" event={"ID":"5ddc00d3-f8dc-4f9f-883c-a221f9062c18","Type":"ContainerDied","Data":"efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c"}
Sep 15 17:00:10.324182 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:10.324161    2566 generic.go:356] "Generic (PLEG): container finished" podID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerID="a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d" exitCode=0
Sep 15 17:00:10.324279 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:10.324219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" event={"ID":"6fe76ae1-caba-4adc-8998-cac717fcabe0","Type":"ContainerDied","Data":"a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d"}
Sep 15 17:00:11.331402 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:11.331366    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" event={"ID":"6fe76ae1-caba-4adc-8998-cac717fcabe0","Type":"ContainerStarted","Data":"5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22"}
Sep 15 17:00:11.331895 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:11.331438    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" event={"ID":"6fe76ae1-caba-4adc-8998-cac717fcabe0","Type":"ContainerStarted","Data":"35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f"}
Sep 15 17:00:11.331895 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:11.331673    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:11.333542 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:11.333517    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" event={"ID":"5ddc00d3-f8dc-4f9f-883c-a221f9062c18","Type":"ContainerStarted","Data":"6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b"}
Sep 15 17:00:11.333694 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:11.333595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" event={"ID":"5ddc00d3-f8dc-4f9f-883c-a221f9062c18","Type":"ContainerStarted","Data":"d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779"}
Sep 15 17:00:11.333802 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:11.333785    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:11.349959 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:11.349913    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podStartSLOduration=3.412264666 podStartE2EDuration="7.349899523s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.937634857s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:00:11.349244528 +0000 UTC m=+1999.717069739" watchObservedRunningTime="2026-09-15 17:00:11.349899523 +0000 UTC m=+1999.717724722"
Sep 15 17:00:11.367342 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:11.367291    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podStartSLOduration=3.4203454300000002 podStartE2EDuration="7.367276607s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.946931177s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:00:11.3662363 +0000 UTC m=+1999.734061500" watchObservedRunningTime="2026-09-15 17:00:11.367276607 +0000 UTC m=+1999.735101806"
Sep 15 17:00:12.338489 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:12.338452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:12.338947 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:12.338519    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:12.339935 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:12.339906    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:00:12.339935 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:12.339918    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:00:13.343280 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:13.343233    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:00:13.343700 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:13.343339    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:00:18.348762 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:18.348731    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:00:18.349179 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:18.348819    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:00:18.349299 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:18.349269    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:00:18.349637 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:18.349604    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:00:28.349566 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:28.349521    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:00:28.349566 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:28.349531    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:00:38.350053 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:38.350010    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:00:38.350641 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:38.350010    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:00:48.349501 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:48.349458    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:00:48.349901 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:48.349619    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:00:58.349631 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:58.349589    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:00:58.350090 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:00:58.349595    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:01:08.349933 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:08.349890    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:01:08.350457 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:08.349897    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:01:18.349682 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:18.349594    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:01:18.349682 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:18.349616    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:01:28.350718 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:28.350688    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:01:28.351153 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:28.350777    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:01:34.900631 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:34.900593    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"]
Sep 15 17:01:34.901113 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:34.900880    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kube-rbac-proxy" containerID="cri-o://6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b" gracePeriod=30
Sep 15 17:01:34.901113 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:34.900874    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" containerID="cri-o://d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779" gracePeriod=30
Sep 15 17:01:35.729069 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:35.729037    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerID="6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b" exitCode=2
Sep 15 17:01:35.729234 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:35.729081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" event={"ID":"5ddc00d3-f8dc-4f9f-883c-a221f9062c18","Type":"ContainerDied","Data":"6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b"}
Sep 15 17:01:38.344334 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:38.344288    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.46:8643/healthz\": dial tcp 10.132.0.46:8643: connect: connection refused"
Sep 15 17:01:38.349640 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:38.349605    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.46:8080: connect: connection refused"
Sep 15 17:01:39.448670 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.448648    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:01:39.601904 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.601836    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-rbac-proxy-sar-config\") pod \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") "
Sep 15 17:01:39.601904 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.601873    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rz2hr\" (UniqueName: \"kubernetes.io/projected/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-api-access-rz2hr\") pod \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") "
Sep 15 17:01:39.601904 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.601901    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-cabundle-cert\") pod \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") "
Sep 15 17:01:39.602102 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.602012    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls\") pod \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") "
Sep 15 17:01:39.602102 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.602074    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kserve-provision-location\") pod \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\" (UID: \"5ddc00d3-f8dc-4f9f-883c-a221f9062c18\") "
Sep 15 17:01:39.602279 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.602253    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5ddc00d3-f8dc-4f9f-883c-a221f9062c18" (UID: "5ddc00d3-f8dc-4f9f-883c-a221f9062c18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:01:39.602279 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.602258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5ddc00d3-f8dc-4f9f-883c-a221f9062c18" (UID: "5ddc00d3-f8dc-4f9f-883c-a221f9062c18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:01:39.602444 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.602422    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5ddc00d3-f8dc-4f9f-883c-a221f9062c18" (UID: "5ddc00d3-f8dc-4f9f-883c-a221f9062c18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:01:39.603971 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.603949    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5ddc00d3-f8dc-4f9f-883c-a221f9062c18" (UID: "5ddc00d3-f8dc-4f9f-883c-a221f9062c18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:01:39.604054 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.604035    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-api-access-rz2hr" (OuterVolumeSpecName: "kube-api-access-rz2hr") pod "5ddc00d3-f8dc-4f9f-883c-a221f9062c18" (UID: "5ddc00d3-f8dc-4f9f-883c-a221f9062c18"). InnerVolumeSpecName "kube-api-access-rz2hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:01:39.703125 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.703093    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:01:39.703125 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.703122    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rz2hr\" (UniqueName: \"kubernetes.io/projected/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kube-api-access-rz2hr\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:01:39.703125 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.703132    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:01:39.703297 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.703140    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:01:39.703297 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.703148    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5ddc00d3-f8dc-4f9f-883c-a221f9062c18-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:01:39.748671 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.748638    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerID="d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779" exitCode=0
Sep 15 17:01:39.748772 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.748705    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"
Sep 15 17:01:39.748772 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.748722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" event={"ID":"5ddc00d3-f8dc-4f9f-883c-a221f9062c18","Type":"ContainerDied","Data":"d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779"}
Sep 15 17:01:39.748849 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.748782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9" event={"ID":"5ddc00d3-f8dc-4f9f-883c-a221f9062c18","Type":"ContainerDied","Data":"26b34b6b383641246a604987ec4aa6e906892eb8dcd1e4f56d9c145900f070d4"}
Sep 15 17:01:39.748849 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.748810    2566 scope.go:117] "RemoveContainer" containerID="6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b"
Sep 15 17:01:39.757349 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.757333    2566 scope.go:117] "RemoveContainer" containerID="d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779"
Sep 15 17:01:39.764670 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.764649    2566 scope.go:117] "RemoveContainer" containerID="efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c"
Sep 15 17:01:39.770167 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.770147    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"]
Sep 15 17:01:39.772497 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.772480    2566 scope.go:117] "RemoveContainer" containerID="6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b"
Sep 15 17:01:39.772914 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:01:39.772893    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b\": container with ID starting with 6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b not found: ID does not exist" containerID="6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b"
Sep 15 17:01:39.773012 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.772922    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b"} err="failed to get container status \"6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b\": rpc error: code = NotFound desc = could not find container \"6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b\": container with ID starting with 6e7d7b33891d4ceb907d930ee74be0dff6666710cec4d8654cd9c9f494776d4b not found: ID does not exist"
Sep 15 17:01:39.773012 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.772943    2566 scope.go:117] "RemoveContainer" containerID="d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779"
Sep 15 17:01:39.773247 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:01:39.773227    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779\": container with ID starting with d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779 not found: ID does not exist" containerID="d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779"
Sep 15 17:01:39.773327 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.773256    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779"} err="failed to get container status \"d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779\": rpc error: code = NotFound desc = could not find container \"d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779\": container with ID starting with d3175b0e23e3b0bfb865174b356eaedf613c81ec4bdc4c8cf41ae42264863779 not found: ID does not exist"
Sep 15 17:01:39.773327 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.773280    2566 scope.go:117] "RemoveContainer" containerID="efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c"
Sep 15 17:01:39.773550 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:01:39.773532    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c\": container with ID starting with efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c not found: ID does not exist" containerID="efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c"
Sep 15 17:01:39.773620 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.773558    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c"} err="failed to get container status \"efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c\": rpc error: code = NotFound desc = could not find container \"efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c\": container with ID starting with efca758640f8dfbcac95a0859ba9882c189fc210da83021ebde90429d408d26c not found: ID does not exist"
Sep 15 17:01:39.774202 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:39.774185    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-6bf7c9cd7c-mkkt9"]
Sep 15 17:01:40.159013 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:40.158981    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" path="/var/lib/kubelet/pods/5ddc00d3-f8dc-4f9f-883c-a221f9062c18/volumes"
Sep 15 17:01:52.207305 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:52.207274    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:01:52.218248 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:01:52.218227    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:03:15.060454 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:15.060369    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"]
Sep 15 17:03:15.061010 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:15.060693    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" containerID="cri-o://35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f" gracePeriod=30
Sep 15 17:03:15.061010 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:15.060761    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kube-rbac-proxy" containerID="cri-o://5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22" gracePeriod=30
Sep 15 17:03:15.183065 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:15.183035    2566 generic.go:356] "Generic (PLEG): container finished" podID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerID="5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22" exitCode=2
Sep 15 17:03:15.183190 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:15.183103    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" event={"ID":"6fe76ae1-caba-4adc-8998-cac717fcabe0","Type":"ContainerDied","Data":"5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22"}
Sep 15 17:03:18.343934 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:18.343888    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.47:8643/healthz\": dial tcp 10.132.0.47:8643: connect: connection refused"
Sep 15 17:03:18.350083 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:18.350056    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.47:8080: connect: connection refused"
Sep 15 17:03:19.712465 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.712435    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:03:19.816131 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.816050    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe76ae1-caba-4adc-8998-cac717fcabe0-proxy-tls\") pod \"6fe76ae1-caba-4adc-8998-cac717fcabe0\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") "
Sep 15 17:03:19.816131 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.816103    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wp8h\" (UniqueName: \"kubernetes.io/projected/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-api-access-2wp8h\") pod \"6fe76ae1-caba-4adc-8998-cac717fcabe0\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") "
Sep 15 17:03:19.816432 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.816159    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-rbac-proxy-sar-config\") pod \"6fe76ae1-caba-4adc-8998-cac717fcabe0\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") "
Sep 15 17:03:19.816432 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.816195    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-cabundle-cert\") pod \"6fe76ae1-caba-4adc-8998-cac717fcabe0\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") "
Sep 15 17:03:19.816432 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.816251    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe76ae1-caba-4adc-8998-cac717fcabe0-kserve-provision-location\") pod \"6fe76ae1-caba-4adc-8998-cac717fcabe0\" (UID: \"6fe76ae1-caba-4adc-8998-cac717fcabe0\") "
Sep 15 17:03:19.816629 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.816595    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6fe76ae1-caba-4adc-8998-cac717fcabe0" (UID: "6fe76ae1-caba-4adc-8998-cac717fcabe0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:03:19.816696 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.816624    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6fe76ae1-caba-4adc-8998-cac717fcabe0" (UID: "6fe76ae1-caba-4adc-8998-cac717fcabe0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:03:19.816740 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.816680    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fe76ae1-caba-4adc-8998-cac717fcabe0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6fe76ae1-caba-4adc-8998-cac717fcabe0" (UID: "6fe76ae1-caba-4adc-8998-cac717fcabe0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:03:19.818163 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.818143    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe76ae1-caba-4adc-8998-cac717fcabe0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6fe76ae1-caba-4adc-8998-cac717fcabe0" (UID: "6fe76ae1-caba-4adc-8998-cac717fcabe0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:03:19.818285 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.818267    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-api-access-2wp8h" (OuterVolumeSpecName: "kube-api-access-2wp8h") pod "6fe76ae1-caba-4adc-8998-cac717fcabe0" (UID: "6fe76ae1-caba-4adc-8998-cac717fcabe0"). InnerVolumeSpecName "kube-api-access-2wp8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:03:19.917101 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.917066    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:03:19.917101 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.917096    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6fe76ae1-caba-4adc-8998-cac717fcabe0-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:03:19.917304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.917110    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6fe76ae1-caba-4adc-8998-cac717fcabe0-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:03:19.917304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.917122    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6fe76ae1-caba-4adc-8998-cac717fcabe0-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:03:19.917304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:19.917135    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wp8h\" (UniqueName: \"kubernetes.io/projected/6fe76ae1-caba-4adc-8998-cac717fcabe0-kube-api-access-2wp8h\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:03:20.207675 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.207635    2566 generic.go:356] "Generic (PLEG): container finished" podID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerID="35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f" exitCode=0
Sep 15 17:03:20.207904 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.207697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" event={"ID":"6fe76ae1-caba-4adc-8998-cac717fcabe0","Type":"ContainerDied","Data":"35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f"}
Sep 15 17:03:20.207904 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.207706    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"
Sep 15 17:03:20.207904 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.207752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m" event={"ID":"6fe76ae1-caba-4adc-8998-cac717fcabe0","Type":"ContainerDied","Data":"642579b72bbf09dd068b57493e85b9cd4f07da6a75734bcbc541fde8cbfb649b"}
Sep 15 17:03:20.207904 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.207781    2566 scope.go:117] "RemoveContainer" containerID="5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22"
Sep 15 17:03:20.217609 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.217585    2566 scope.go:117] "RemoveContainer" containerID="35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f"
Sep 15 17:03:20.227053 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.227026    2566 scope.go:117] "RemoveContainer" containerID="a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d"
Sep 15 17:03:20.227644 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.227613    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"]
Sep 15 17:03:20.234718 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.234692    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-6b76f97c96-fxx9m"]
Sep 15 17:03:20.238124 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.238100    2566 scope.go:117] "RemoveContainer" containerID="5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22"
Sep 15 17:03:20.238478 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:03:20.238451    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22\": container with ID starting with 5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22 not found: ID does not exist" containerID="5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22"
Sep 15 17:03:20.238603 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.238490    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22"} err="failed to get container status \"5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22\": rpc error: code = NotFound desc = could not find container \"5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22\": container with ID starting with 5fc7191e8c40e50c819443852f0eb3c1debc4a1e76c75b1dfeac4892f80c4b22 not found: ID does not exist"
Sep 15 17:03:20.238603 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.238524    2566 scope.go:117] "RemoveContainer" containerID="35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f"
Sep 15 17:03:20.238807 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.238787    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"]
Sep 15 17:03:20.238971 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:03:20.238950    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f\": container with ID starting with 35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f not found: ID does not exist" containerID="35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f"
Sep 15 17:03:20.239023 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.238978    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f"} err="failed to get container status \"35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f\": rpc error: code = NotFound desc = could not find container \"35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f\": container with ID starting with 35282ba2168410fc1d5ed58bd21555cca494264969c0bf7ba39da9f936eb807f not found: ID does not exist"
Sep 15 17:03:20.239023 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.238998    2566 scope.go:117] "RemoveContainer" containerID="a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d"
Sep 15 17:03:20.239132 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239117    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kube-rbac-proxy"
Sep 15 17:03:20.239186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239137    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kube-rbac-proxy"
Sep 15 17:03:20.239186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239151    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="storage-initializer"
Sep 15 17:03:20.239186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239160    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="storage-initializer"
Sep 15 17:03:20.239186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239171    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container"
Sep 15 17:03:20.239186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239180    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239213    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="storage-initializer"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239223    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="storage-initializer"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239230    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239236    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239253    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kube-rbac-proxy"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239259    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kube-rbac-proxy"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:03:20.239283    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d\": container with ID starting with a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d not found: ID does not exist" containerID="a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239301    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d"} err="failed to get container status \"a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d\": rpc error: code = NotFound desc = could not find container \"a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d\": container with ID starting with a6e9008c626166fbdd7227096dcaa6ad012fafe5f1968ee3efd7dbf5c7e1ff3d not found: ID does not exist"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239327    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kserve-container"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239336    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kube-rbac-proxy"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239341    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" containerName="kube-rbac-proxy"
Sep 15 17:03:20.239356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.239350    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ddc00d3-f8dc-4f9f-883c-a221f9062c18" containerName="kserve-container"
Sep 15 17:03:20.243917 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.243900    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.246382 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.246361    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 15 17:03:20.246500 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.246394    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:03:20.246500 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.246431    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:03:20.246500 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.246440    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:03:20.246706 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.246502    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:03:20.246706 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.246702    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 15 17:03:20.246808 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.246729    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:03:20.252116 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.252094    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"]
Sep 15 17:03:20.321147 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.321103    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7580928c-2053-483f-8b9a-c4c931433bac-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.321350 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.321154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7580928c-2053-483f-8b9a-c4c931433bac-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.321350 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.321185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvw7\" (UniqueName: \"kubernetes.io/projected/7580928c-2053-483f-8b9a-c4c931433bac-kube-api-access-ljvw7\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.321350 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.321226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.321350 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.321308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.323183 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.323156    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"]
Sep 15 17:03:20.326710 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.326692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.328818 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.328789    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 15 17:03:20.337269 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.337249    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"]
Sep 15 17:03:20.422006 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.421927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.422006 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.421973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.422006 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.421996    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.422258 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.422258 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422039    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.422258 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmn9\" (UniqueName: \"kubernetes.io/projected/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-api-access-bwmn9\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.422258 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422146    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7580928c-2053-483f-8b9a-c4c931433bac-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.422258 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7580928c-2053-483f-8b9a-c4c931433bac-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.422258 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422221    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvw7\" (UniqueName: \"kubernetes.io/projected/7580928c-2053-483f-8b9a-c4c931433bac-kube-api-access-ljvw7\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.422258 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b94549a-7e75-4e24-ba50-c86127172f4a-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.422689 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7580928c-2053-483f-8b9a-c4c931433bac-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.422755 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.422755 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.422731    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.424742 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.424723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7580928c-2053-483f-8b9a-c4c931433bac-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.430416 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.430394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvw7\" (UniqueName: \"kubernetes.io/projected/7580928c-2053-483f-8b9a-c4c931433bac-kube-api-access-ljvw7\") pod \"isvc-canary-stop-v2-predictor-6d74b8779c-58ktw\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.522882 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.522854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b94549a-7e75-4e24-ba50-c86127172f4a-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.523027 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.522899    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.523027 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.522922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.523027 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.522943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.523027 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:03:20.523020    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 15 17:03:20.523254 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.523053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmn9\" (UniqueName: \"kubernetes.io/projected/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-api-access-bwmn9\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.523254 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:03:20.523072    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls podName:4b94549a-7e75-4e24-ba50-c86127172f4a nodeName:}" failed. No retries permitted until 2026-09-15 17:03:21.023053927 +0000 UTC m=+2189.390879118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls") pod "isvc-canary-stop-predictor-7f6f9c568d-xfg42" (UID: "4b94549a-7e75-4e24-ba50-c86127172f4a") : secret "isvc-canary-stop-predictor-serving-cert" not found
Sep 15 17:03:20.523372 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.523256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b94549a-7e75-4e24-ba50-c86127172f4a-kserve-provision-location\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.523634 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.523612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.523728 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.523662    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-cabundle-cert\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.530787 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.530765    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmn9\" (UniqueName: \"kubernetes.io/projected/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-api-access-bwmn9\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:20.556180 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.556159    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:20.833939 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:20.833901    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"]
Sep 15 17:03:20.836410 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:03:20.836383    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7580928c_2053_483f_8b9a_c4c931433bac.slice/crio-3ad0f8135cd5e77403cf6856b406b0f0e98a65487141e65822e894ba22ec1235 WatchSource:0}: Error finding container 3ad0f8135cd5e77403cf6856b406b0f0e98a65487141e65822e894ba22ec1235: Status 404 returned error can't find the container with id 3ad0f8135cd5e77403cf6856b406b0f0e98a65487141e65822e894ba22ec1235
Sep 15 17:03:21.026629 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:21.026594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:21.029086 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:21.029051    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls\") pod \"isvc-canary-stop-predictor-7f6f9c568d-xfg42\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:21.214318 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:21.214276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" event={"ID":"7580928c-2053-483f-8b9a-c4c931433bac","Type":"ContainerStarted","Data":"10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5"}
Sep 15 17:03:21.214491 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:21.214339    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" event={"ID":"7580928c-2053-483f-8b9a-c4c931433bac","Type":"ContainerStarted","Data":"3ad0f8135cd5e77403cf6856b406b0f0e98a65487141e65822e894ba22ec1235"}
Sep 15 17:03:21.237501 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:21.237462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:21.511072 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:21.510913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"]
Sep 15 17:03:21.513111 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:03:21.513087    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b94549a_7e75_4e24_ba50_c86127172f4a.slice/crio-9527bb1a3e0d92c42d06e55d72fe833e77966c1b6e6373c70d57b816fcd07153 WatchSource:0}: Error finding container 9527bb1a3e0d92c42d06e55d72fe833e77966c1b6e6373c70d57b816fcd07153: Status 404 returned error can't find the container with id 9527bb1a3e0d92c42d06e55d72fe833e77966c1b6e6373c70d57b816fcd07153
Sep 15 17:03:22.159994 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:22.159960    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fe76ae1-caba-4adc-8998-cac717fcabe0" path="/var/lib/kubelet/pods/6fe76ae1-caba-4adc-8998-cac717fcabe0/volumes"
Sep 15 17:03:22.220099 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:22.220064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" event={"ID":"4b94549a-7e75-4e24-ba50-c86127172f4a","Type":"ContainerStarted","Data":"8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce"}
Sep 15 17:03:22.220280 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:22.220136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" event={"ID":"4b94549a-7e75-4e24-ba50-c86127172f4a","Type":"ContainerStarted","Data":"9527bb1a3e0d92c42d06e55d72fe833e77966c1b6e6373c70d57b816fcd07153"}
Sep 15 17:03:25.234483 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:25.234448    2566 generic.go:356] "Generic (PLEG): container finished" podID="7580928c-2053-483f-8b9a-c4c931433bac" containerID="10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5" exitCode=0
Sep 15 17:03:25.234881 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:25.234526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" event={"ID":"7580928c-2053-483f-8b9a-c4c931433bac","Type":"ContainerDied","Data":"10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5"}
Sep 15 17:03:26.241797 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:26.241763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" event={"ID":"7580928c-2053-483f-8b9a-c4c931433bac","Type":"ContainerStarted","Data":"7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c"}
Sep 15 17:03:26.242201 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:26.241831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" event={"ID":"7580928c-2053-483f-8b9a-c4c931433bac","Type":"ContainerStarted","Data":"75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940"}
Sep 15 17:03:26.242201 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:26.242113    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:26.242201 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:26.242172    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:26.243460 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:26.243436    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerID="8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce" exitCode=0
Sep 15 17:03:26.243591 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:26.243465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" event={"ID":"4b94549a-7e75-4e24-ba50-c86127172f4a","Type":"ContainerDied","Data":"8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce"}
Sep 15 17:03:26.243791 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:26.243765    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:03:26.261407 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:26.261353    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podStartSLOduration=1.978571807 podStartE2EDuration="6.261335877s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.28276407s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:03:26.259960074 +0000 UTC m=+2194.627785284" watchObservedRunningTime="2026-09-15 17:03:26.261335877 +0000 UTC m=+2194.629161077"
Sep 15 17:03:27.250765 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:27.250727    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" event={"ID":"4b94549a-7e75-4e24-ba50-c86127172f4a","Type":"ContainerStarted","Data":"a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3"}
Sep 15 17:03:27.251251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:27.250789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" event={"ID":"4b94549a-7e75-4e24-ba50-c86127172f4a","Type":"ContainerStarted","Data":"ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab"}
Sep 15 17:03:27.251251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:27.251072    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:03:27.251251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:27.251144    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:27.251251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:27.251197    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:27.252725 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:27.252700    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:03:27.270208 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:27.270155    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podStartSLOduration=3.367186221 podStartE2EDuration="7.270139041s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.90295282s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:03:27.268465853 +0000 UTC m=+2195.636291077" watchObservedRunningTime="2026-09-15 17:03:27.270139041 +0000 UTC m=+2195.637964240"
Sep 15 17:03:28.255087 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:28.255023    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:03:32.256586 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:32.256535    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:03:32.257135 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:32.257102    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:03:33.259722 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:33.259677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:03:33.260416 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:33.260390    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:03:42.257391 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:42.257348    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:03:43.260813 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:43.260752    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:03:52.258043 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:52.257999    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:03:53.261087 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:03:53.261050    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:04:02.257613 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:02.257551    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:04:03.260332 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:03.260289    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:04:12.257592 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:12.257536    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:04:13.260317 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:13.260277    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:04:22.257042 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:22.256955    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:04:23.260840 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:23.260801    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:04:32.257969 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:32.257929    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:04:33.260858 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:33.260813    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:04:42.159245 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:42.159215    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:04:42.258442 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:42.258406    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:04:50.251398 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.251361    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"]
Sep 15 17:04:50.251948 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.251686    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" containerID="cri-o://75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940" gracePeriod=30
Sep 15 17:04:50.251948 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.251734    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kube-rbac-proxy" containerID="cri-o://7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c" gracePeriod=30
Sep 15 17:04:50.281382 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.281349    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"]
Sep 15 17:04:50.281681 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.281623    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" containerID="cri-o://ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab" gracePeriod=30
Sep 15 17:04:50.281840 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.281684    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kube-rbac-proxy" containerID="cri-o://a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3" gracePeriod=30
Sep 15 17:04:50.641834 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.641802    2566 generic.go:356] "Generic (PLEG): container finished" podID="7580928c-2053-483f-8b9a-c4c931433bac" containerID="7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c" exitCode=2
Sep 15 17:04:50.642001 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.641876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" event={"ID":"7580928c-2053-483f-8b9a-c4c931433bac","Type":"ContainerDied","Data":"7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c"}
Sep 15 17:04:50.644019 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.643997    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerID="a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3" exitCode=2
Sep 15 17:04:50.644140 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:50.644045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" event={"ID":"4b94549a-7e75-4e24-ba50-c86127172f4a","Type":"ContainerDied","Data":"a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3"}
Sep 15 17:04:52.165857 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:52.165818    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.49:8080: connect: connection refused"
Sep 15 17:04:52.251841 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:52.251799    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.48:8643/healthz\": dial tcp 10.132.0.48:8643: connect: connection refused"
Sep 15 17:04:52.257073 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:52.257030    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.48:8080: connect: connection refused"
Sep 15 17:04:53.256124 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:53.256070    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.49:8643/healthz\": dial tcp 10.132.0.49:8643: connect: connection refused"
Sep 15 17:04:55.116027 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.115965    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:04:55.237164 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237134    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"]
Sep 15 17:04:55.237633 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237615    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="storage-initializer"
Sep 15 17:04:55.237676 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237636    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="storage-initializer"
Sep 15 17:04:55.237676 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237660    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kube-rbac-proxy"
Sep 15 17:04:55.237676 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237668    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kube-rbac-proxy"
Sep 15 17:04:55.237787 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237685    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container"
Sep 15 17:04:55.237787 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237693    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container"
Sep 15 17:04:55.237858 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237792    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kube-rbac-proxy"
Sep 15 17:04:55.237858 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.237805    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7580928c-2053-483f-8b9a-c4c931433bac" containerName="kserve-container"
Sep 15 17:04:55.241540 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.241517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.244210 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.244182    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 15 17:04:55.244304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.244221    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 15 17:04:55.247710 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.247686    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7580928c-2053-483f-8b9a-c4c931433bac-kserve-provision-location\") pod \"7580928c-2053-483f-8b9a-c4c931433bac\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") "
Sep 15 17:04:55.247816 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.247718    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-kube-rbac-proxy-sar-config\") pod \"7580928c-2053-483f-8b9a-c4c931433bac\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") "
Sep 15 17:04:55.247816 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.247759    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-cabundle-cert\") pod \"7580928c-2053-483f-8b9a-c4c931433bac\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") "
Sep 15 17:04:55.247929 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.247821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljvw7\" (UniqueName: \"kubernetes.io/projected/7580928c-2053-483f-8b9a-c4c931433bac-kube-api-access-ljvw7\") pod \"7580928c-2053-483f-8b9a-c4c931433bac\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") "
Sep 15 17:04:55.247929 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.247848    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7580928c-2053-483f-8b9a-c4c931433bac-proxy-tls\") pod \"7580928c-2053-483f-8b9a-c4c931433bac\" (UID: \"7580928c-2053-483f-8b9a-c4c931433bac\") "
Sep 15 17:04:55.248068 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.248042    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7580928c-2053-483f-8b9a-c4c931433bac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7580928c-2053-483f-8b9a-c4c931433bac" (UID: "7580928c-2053-483f-8b9a-c4c931433bac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:04:55.248194 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.248170    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7580928c-2053-483f-8b9a-c4c931433bac" (UID: "7580928c-2053-483f-8b9a-c4c931433bac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:55.248260 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.248206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7580928c-2053-483f-8b9a-c4c931433bac" (UID: "7580928c-2053-483f-8b9a-c4c931433bac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:55.251251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.251224    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7580928c-2053-483f-8b9a-c4c931433bac-kube-api-access-ljvw7" (OuterVolumeSpecName: "kube-api-access-ljvw7") pod "7580928c-2053-483f-8b9a-c4c931433bac" (UID: "7580928c-2053-483f-8b9a-c4c931433bac"). InnerVolumeSpecName "kube-api-access-ljvw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:04:55.251563 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.251534    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7580928c-2053-483f-8b9a-c4c931433bac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7580928c-2053-483f-8b9a-c4c931433bac" (UID: "7580928c-2053-483f-8b9a-c4c931433bac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:04:55.252549 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.252527    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"]
Sep 15 17:04:55.254309 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.254217    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:04:55.348995 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.348962    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwmn9\" (UniqueName: \"kubernetes.io/projected/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-api-access-bwmn9\") pod \"4b94549a-7e75-4e24-ba50-c86127172f4a\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") "
Sep 15 17:04:55.349170 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349057    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b94549a-7e75-4e24-ba50-c86127172f4a-kserve-provision-location\") pod \"4b94549a-7e75-4e24-ba50-c86127172f4a\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") "
Sep 15 17:04:55.349170 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349107    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls\") pod \"4b94549a-7e75-4e24-ba50-c86127172f4a\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") "
Sep 15 17:04:55.349170 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349124    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-cabundle-cert\") pod \"4b94549a-7e75-4e24-ba50-c86127172f4a\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") "
Sep 15 17:04:55.349170 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349141    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-rbac-proxy-sar-config\") pod \"4b94549a-7e75-4e24-ba50-c86127172f4a\" (UID: \"4b94549a-7e75-4e24-ba50-c86127172f4a\") "
Sep 15 17:04:55.349402 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.349402 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349235    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.349402 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.349402 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rthmf\" (UniqueName: \"kubernetes.io/projected/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-api-access-rthmf\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.349402 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349425    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b94549a-7e75-4e24-ba50-c86127172f4a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b94549a-7e75-4e24-ba50-c86127172f4a" (UID: "4b94549a-7e75-4e24-ba50-c86127172f4a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349483    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b94549a-7e75-4e24-ba50-c86127172f4a" (UID: "4b94549a-7e75-4e24-ba50-c86127172f4a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349507    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349524    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljvw7\" (UniqueName: \"kubernetes.io/projected/7580928c-2053-483f-8b9a-c4c931433bac-kube-api-access-ljvw7\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349539    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7580928c-2053-483f-8b9a-c4c931433bac-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349536    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b94549a-7e75-4e24-ba50-c86127172f4a" (UID: "4b94549a-7e75-4e24-ba50-c86127172f4a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349553    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b94549a-7e75-4e24-ba50-c86127172f4a-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349565    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7580928c-2053-483f-8b9a-c4c931433bac-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.349648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.349605    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7580928c-2053-483f-8b9a-c4c931433bac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.351639 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.351616    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b94549a-7e75-4e24-ba50-c86127172f4a" (UID: "4b94549a-7e75-4e24-ba50-c86127172f4a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:04:55.351639 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.351628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-api-access-bwmn9" (OuterVolumeSpecName: "kube-api-access-bwmn9") pod "4b94549a-7e75-4e24-ba50-c86127172f4a" (UID: "4b94549a-7e75-4e24-ba50-c86127172f4a"). InnerVolumeSpecName "kube-api-access-bwmn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:04:55.450613 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.450613 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rthmf\" (UniqueName: \"kubernetes.io/projected/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-api-access-rthmf\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450759    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b94549a-7e75-4e24-ba50-c86127172f4a-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450776    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450790    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:04:55.450788    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-serving-cert: secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.450804    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmn9\" (UniqueName: \"kubernetes.io/projected/4b94549a-7e75-4e24-ba50-c86127172f4a-kube-api-access-bwmn9\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:04:55.450896 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:04:55.450870    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls podName:9b332cd3-f2b6-4349-ab85-7d5989a797d8 nodeName:}" failed. No retries permitted until 2026-09-15 17:04:55.950846776 +0000 UTC m=+2284.318671974 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls") pod "isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" (UID: "9b332cd3-f2b6-4349-ab85-7d5989a797d8") : secret "isvc-xgboost-runtime-predictor-serving-cert" not found
Sep 15 17:04:55.451318 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.451133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.451318 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.451271    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.451388 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.451334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.459127 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.459101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rthmf\" (UniqueName: \"kubernetes.io/projected/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-api-access-rthmf\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.669103 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.669067    2566 generic.go:356] "Generic (PLEG): container finished" podID="7580928c-2053-483f-8b9a-c4c931433bac" containerID="75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940" exitCode=0
Sep 15 17:04:55.669276 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.669131    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"
Sep 15 17:04:55.669276 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.669151    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" event={"ID":"7580928c-2053-483f-8b9a-c4c931433bac","Type":"ContainerDied","Data":"75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940"}
Sep 15 17:04:55.669276 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.669219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw" event={"ID":"7580928c-2053-483f-8b9a-c4c931433bac","Type":"ContainerDied","Data":"3ad0f8135cd5e77403cf6856b406b0f0e98a65487141e65822e894ba22ec1235"}
Sep 15 17:04:55.669276 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.669260    2566 scope.go:117] "RemoveContainer" containerID="7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c"
Sep 15 17:04:55.671348 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.671325    2566 generic.go:356] "Generic (PLEG): container finished" podID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerID="ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab" exitCode=0
Sep 15 17:04:55.671491 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.671390    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"
Sep 15 17:04:55.671491 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.671402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" event={"ID":"4b94549a-7e75-4e24-ba50-c86127172f4a","Type":"ContainerDied","Data":"ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab"}
Sep 15 17:04:55.671491 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.671465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42" event={"ID":"4b94549a-7e75-4e24-ba50-c86127172f4a","Type":"ContainerDied","Data":"9527bb1a3e0d92c42d06e55d72fe833e77966c1b6e6373c70d57b816fcd07153"}
Sep 15 17:04:55.678385 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.678272    2566 scope.go:117] "RemoveContainer" containerID="75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940"
Sep 15 17:04:55.686065 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.686050    2566 scope.go:117] "RemoveContainer" containerID="10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5"
Sep 15 17:04:55.693258 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.693234    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"]
Sep 15 17:04:55.693678 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.693661    2566 scope.go:117] "RemoveContainer" containerID="7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c"
Sep 15 17:04:55.693928 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:04:55.693908    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c\": container with ID starting with 7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c not found: ID does not exist" containerID="7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c"
Sep 15 17:04:55.694010 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.693943    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c"} err="failed to get container status \"7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c\": rpc error: code = NotFound desc = could not find container \"7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c\": container with ID starting with 7395f104ba78916895edfbe283c5a808f04725347d7b638a4a98896144cdfa7c not found: ID does not exist"
Sep 15 17:04:55.694010 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.693966    2566 scope.go:117] "RemoveContainer" containerID="75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940"
Sep 15 17:04:55.694433 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:04:55.694320    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940\": container with ID starting with 75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940 not found: ID does not exist" containerID="75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940"
Sep 15 17:04:55.694433 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.694350    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940"} err="failed to get container status \"75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940\": rpc error: code = NotFound desc = could not find container \"75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940\": container with ID starting with 75394bcd436d558383032af6e3c985b867127a2838ef42e3b4deae190a2bc940 not found: ID does not exist"
Sep 15 17:04:55.694433 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.694371    2566 scope.go:117] "RemoveContainer" containerID="10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5"
Sep 15 17:04:55.694820 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:04:55.694786    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5\": container with ID starting with 10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5 not found: ID does not exist" containerID="10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5"
Sep 15 17:04:55.694897 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.694815    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5"} err="failed to get container status \"10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5\": rpc error: code = NotFound desc = could not find container \"10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5\": container with ID starting with 10afe444036b463e7824c1b6dde08d002f001f59b07d2eaad535d09a96dc2ca5 not found: ID does not exist"
Sep 15 17:04:55.694897 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.694834    2566 scope.go:117] "RemoveContainer" containerID="a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3"
Sep 15 17:04:55.696730 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.696712    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-6d74b8779c-58ktw"]
Sep 15 17:04:55.702155 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.702140    2566 scope.go:117] "RemoveContainer" containerID="ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab"
Sep 15 17:04:55.709728 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.707630    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"]
Sep 15 17:04:55.711629 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.711609    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-7f6f9c568d-xfg42"]
Sep 15 17:04:55.713243 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.713226    2566 scope.go:117] "RemoveContainer" containerID="8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce"
Sep 15 17:04:55.721634 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.721608    2566 scope.go:117] "RemoveContainer" containerID="a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3"
Sep 15 17:04:55.721985 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:04:55.721960    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3\": container with ID starting with a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3 not found: ID does not exist" containerID="a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3"
Sep 15 17:04:55.722085 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.721994    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3"} err="failed to get container status \"a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3\": rpc error: code = NotFound desc = could not find container \"a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3\": container with ID starting with a7a0148903155acb5286085fd1268b68d91b612bd0eda083cb65bfaa16ebe8a3 not found: ID does not exist"
Sep 15 17:04:55.722085 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.722018    2566 scope.go:117] "RemoveContainer" containerID="ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab"
Sep 15 17:04:55.722324 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:04:55.722293    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab\": container with ID starting with ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab not found: ID does not exist" containerID="ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab"
Sep 15 17:04:55.722407 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.722323    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab"} err="failed to get container status \"ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab\": rpc error: code = NotFound desc = could not find container \"ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab\": container with ID starting with ed66dd9353fd7d2944b1255dbea4c5aa085110f6f8f75124bac69722f4b826ab not found: ID does not exist"
Sep 15 17:04:55.722407 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.722347    2566 scope.go:117] "RemoveContainer" containerID="8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce"
Sep 15 17:04:55.722692 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:04:55.722669    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce\": container with ID starting with 8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce not found: ID does not exist" containerID="8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce"
Sep 15 17:04:55.722764 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.722699    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce"} err="failed to get container status \"8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce\": rpc error: code = NotFound desc = could not find container \"8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce\": container with ID starting with 8ad98f9aeb339bb5c59efb34c6dc6189035719da904d58efb8969100cbe934ce not found: ID does not exist"
Sep 15 17:04:55.955957 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.955835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:55.958276 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:55.958256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-tv2gx\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:56.159261 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:56.159226    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" path="/var/lib/kubelet/pods/4b94549a-7e75-4e24-ba50-c86127172f4a/volumes"
Sep 15 17:04:56.159261 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:56.159254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:04:56.159758 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:56.159743    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7580928c-2053-483f-8b9a-c4c931433bac" path="/var/lib/kubelet/pods/7580928c-2053-483f-8b9a-c4c931433bac/volumes"
Sep 15 17:04:56.430433 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:56.430411    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"]
Sep 15 17:04:56.432612 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:04:56.432585    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b332cd3_f2b6_4349_ab85_7d5989a797d8.slice/crio-7a8016d5057e74629c9c075a87cb52739b79dbbabb5b31781296fd7d6cd09988 WatchSource:0}: Error finding container 7a8016d5057e74629c9c075a87cb52739b79dbbabb5b31781296fd7d6cd09988: Status 404 returned error can't find the container with id 7a8016d5057e74629c9c075a87cb52739b79dbbabb5b31781296fd7d6cd09988
Sep 15 17:04:56.678654 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:56.678614    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" event={"ID":"9b332cd3-f2b6-4349-ab85-7d5989a797d8","Type":"ContainerStarted","Data":"fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2"}
Sep 15 17:04:56.678847 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:04:56.678692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" event={"ID":"9b332cd3-f2b6-4349-ab85-7d5989a797d8","Type":"ContainerStarted","Data":"7a8016d5057e74629c9c075a87cb52739b79dbbabb5b31781296fd7d6cd09988"}
Sep 15 17:05:00.502188 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502107    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"]
Sep 15 17:05:00.502545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502444    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container"
Sep 15 17:05:00.502545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502456    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container"
Sep 15 17:05:00.502545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502463    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="storage-initializer"
Sep 15 17:05:00.502545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502470    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="storage-initializer"
Sep 15 17:05:00.502545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502478    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kube-rbac-proxy"
Sep 15 17:05:00.502545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502484    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kube-rbac-proxy"
Sep 15 17:05:00.502545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502547    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kube-rbac-proxy"
Sep 15 17:05:00.502825 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.502558    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b94549a-7e75-4e24-ba50-c86127172f4a" containerName="kserve-container"
Sep 15 17:05:00.505648 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.505629    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.508033 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.508011    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Sep 15 17:05:00.508144 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.508128    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Sep 15 17:05:00.515815 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.515790    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"]
Sep 15 17:05:00.596612 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.596560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.596748 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.596626    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6tm5\" (UniqueName: \"kubernetes.io/projected/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-api-access-k6tm5\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.596748 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.596648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.596748 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.596682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.596748 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.596708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.697660 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.697624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.697858 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.697669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.697858 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.697751    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.697858 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.697786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6tm5\" (UniqueName: \"kubernetes.io/projected/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-api-access-k6tm5\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.697858 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.697811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.697858 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:05:00.697848    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 15 17:05:00.698132 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:05:00.697915    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls podName:8c5a0bb5-d214-47b3-a1a7-278b09dde64e nodeName:}" failed. No retries permitted until 2026-09-15 17:05:01.197895784 +0000 UTC m=+2289.565720969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-sbd72" (UID: "8c5a0bb5-d214-47b3-a1a7-278b09dde64e") : secret "isvc-lightgbm-predictor-serving-cert" not found
Sep 15 17:05:00.698202 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.698128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.698484 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.698457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.698484 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.698467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:00.699225 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.699205    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerID="fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2" exitCode=0
Sep 15 17:05:00.699293 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.699273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" event={"ID":"9b332cd3-f2b6-4349-ab85-7d5989a797d8","Type":"ContainerDied","Data":"fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2"}
Sep 15 17:05:00.700418 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.700405    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 17:05:00.706371 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:00.706349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6tm5\" (UniqueName: \"kubernetes.io/projected/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-api-access-k6tm5\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:01.202876 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:01.202838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:01.206347 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:01.206316    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-sbd72\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:01.416718 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:01.416680    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:02.260304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:02.260274    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"]
Sep 15 17:05:02.263076 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:05:02.263047    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c5a0bb5_d214_47b3_a1a7_278b09dde64e.slice/crio-472572b7b1e2175cd48a4ced703e65ff55c90295335bf1e8e7073f9b5d99fe66 WatchSource:0}: Error finding container 472572b7b1e2175cd48a4ced703e65ff55c90295335bf1e8e7073f9b5d99fe66: Status 404 returned error can't find the container with id 472572b7b1e2175cd48a4ced703e65ff55c90295335bf1e8e7073f9b5d99fe66
Sep 15 17:05:02.712719 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:02.712684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" event={"ID":"8c5a0bb5-d214-47b3-a1a7-278b09dde64e","Type":"ContainerStarted","Data":"c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4"}
Sep 15 17:05:02.712881 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:02.712745    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" event={"ID":"8c5a0bb5-d214-47b3-a1a7-278b09dde64e","Type":"ContainerStarted","Data":"472572b7b1e2175cd48a4ced703e65ff55c90295335bf1e8e7073f9b5d99fe66"}
Sep 15 17:05:06.740803 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:06.740764    2566 generic.go:356] "Generic (PLEG): container finished" podID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerID="c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4" exitCode=0
Sep 15 17:05:06.741273 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:06.740832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" event={"ID":"8c5a0bb5-d214-47b3-a1a7-278b09dde64e","Type":"ContainerDied","Data":"c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4"}
Sep 15 17:05:21.821007 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.820971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" event={"ID":"9b332cd3-f2b6-4349-ab85-7d5989a797d8","Type":"ContainerStarted","Data":"35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7"}
Sep 15 17:05:21.821495 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.821036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" event={"ID":"9b332cd3-f2b6-4349-ab85-7d5989a797d8","Type":"ContainerStarted","Data":"32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e"}
Sep 15 17:05:21.821495 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.821339    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:05:21.821495 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.821399    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:05:21.823267 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.823234    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:05:21.823378 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.823316    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" event={"ID":"8c5a0bb5-d214-47b3-a1a7-278b09dde64e","Type":"ContainerStarted","Data":"124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a"}
Sep 15 17:05:21.823442 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.823376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" event={"ID":"8c5a0bb5-d214-47b3-a1a7-278b09dde64e","Type":"ContainerStarted","Data":"c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e"}
Sep 15 17:05:21.823608 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.823592    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:21.840693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.840635    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podStartSLOduration=2.500413385 podStartE2EDuration="26.840621392s" totalImagesPullingTime="20.693926635s" totalInitContainerRuntime="3.646281372s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:04:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:05:21.838167313 +0000 UTC m=+2310.205992511" watchObservedRunningTime="2026-09-15 17:05:21.840621392 +0000 UTC m=+2310.208446591"
Sep 15 17:05:21.855819 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:21.855781    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podStartSLOduration=3.032632521 podStartE2EDuration="21.855767758s" totalImagesPullingTime="14.554324668s" totalInitContainerRuntime="4.268810569s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:05:21.85481733 +0000 UTC m=+2310.222642530" watchObservedRunningTime="2026-09-15 17:05:21.855767758 +0000 UTC m=+2310.223592957"
Sep 15 17:05:22.827864 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:22.827827    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:22.828255 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:22.827856    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:05:22.829105 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:22.829082    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:05:23.832626 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:23.832562    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:05:27.832619 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:27.832585    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:05:27.833175 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:27.833145    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:05:28.836778 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:28.836747    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:05:28.837329 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:28.837297    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:05:37.833544 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:37.833505    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:05:38.838228 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:38.838190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:05:47.833126 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:47.833024    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:05:48.837652 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:48.837614    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:05:57.833097 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:57.833056    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:05:58.837160 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:05:58.837119    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:06:07.833217 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:07.833166    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:06:08.837751 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:08.837706    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:06:17.833744 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:17.833703    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:06:18.837814 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:18.837767    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:06:27.833772 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:27.833732    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:06:28.837662 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:28.837623    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:06:34.154366 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:34.154324    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:06:35.353101 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:35.353067    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"]
Sep 15 17:06:35.353522 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:35.353325    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" containerID="cri-o://32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e" gracePeriod=30
Sep 15 17:06:35.353522 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:35.353372    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kube-rbac-proxy" containerID="cri-o://35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7" gracePeriod=30
Sep 15 17:06:36.172525 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:36.172490    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerID="35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7" exitCode=2
Sep 15 17:06:36.172718 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:36.172564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" event={"ID":"9b332cd3-f2b6-4349-ab85-7d5989a797d8","Type":"ContainerDied","Data":"35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7"}
Sep 15 17:06:37.827987 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:37.827946    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.50:8643/healthz\": dial tcp 10.132.0.50:8643: connect: connection refused"
Sep 15 17:06:37.833276 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:37.833251    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.50:8080: connect: connection refused"
Sep 15 17:06:39.005277 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.005249    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:06:39.156031 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156004    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-rbac-proxy-sar-config\") pod \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") "
Sep 15 17:06:39.156191 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156045    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-cabundle-cert\") pod \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") "
Sep 15 17:06:39.156191 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156105    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kserve-provision-location\") pod \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") "
Sep 15 17:06:39.156191 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156129    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rthmf\" (UniqueName: \"kubernetes.io/projected/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-api-access-rthmf\") pod \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") "
Sep 15 17:06:39.156355 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156263    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls\") pod \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\" (UID: \"9b332cd3-f2b6-4349-ab85-7d5989a797d8\") "
Sep 15 17:06:39.156425 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156400    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b332cd3-f2b6-4349-ab85-7d5989a797d8" (UID: "9b332cd3-f2b6-4349-ab85-7d5989a797d8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:06:39.156467 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156438    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b332cd3-f2b6-4349-ab85-7d5989a797d8" (UID: "9b332cd3-f2b6-4349-ab85-7d5989a797d8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:06:39.156467 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156447    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b332cd3-f2b6-4349-ab85-7d5989a797d8" (UID: "9b332cd3-f2b6-4349-ab85-7d5989a797d8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:06:39.156548 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156537    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:39.156613 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156551    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b332cd3-f2b6-4349-ab85-7d5989a797d8-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:39.156613 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.156560    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:39.158261 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.158238    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-api-access-rthmf" (OuterVolumeSpecName: "kube-api-access-rthmf") pod "9b332cd3-f2b6-4349-ab85-7d5989a797d8" (UID: "9b332cd3-f2b6-4349-ab85-7d5989a797d8"). InnerVolumeSpecName "kube-api-access-rthmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:06:39.158355 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.158329    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b332cd3-f2b6-4349-ab85-7d5989a797d8" (UID: "9b332cd3-f2b6-4349-ab85-7d5989a797d8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:06:39.187838 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.187799    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerID="32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e" exitCode=0
Sep 15 17:06:39.187984 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.187861    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"
Sep 15 17:06:39.187984 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.187859    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" event={"ID":"9b332cd3-f2b6-4349-ab85-7d5989a797d8","Type":"ContainerDied","Data":"32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e"}
Sep 15 17:06:39.188063 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.187996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx" event={"ID":"9b332cd3-f2b6-4349-ab85-7d5989a797d8","Type":"ContainerDied","Data":"7a8016d5057e74629c9c075a87cb52739b79dbbabb5b31781296fd7d6cd09988"}
Sep 15 17:06:39.188063 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.188034    2566 scope.go:117] "RemoveContainer" containerID="35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7"
Sep 15 17:06:39.196938 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.196918    2566 scope.go:117] "RemoveContainer" containerID="32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e"
Sep 15 17:06:39.204740 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.204713    2566 scope.go:117] "RemoveContainer" containerID="fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2"
Sep 15 17:06:39.209878 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.209852    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"]
Sep 15 17:06:39.212871 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.212846    2566 scope.go:117] "RemoveContainer" containerID="35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7"
Sep 15 17:06:39.213105 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.213086    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-tv2gx"]
Sep 15 17:06:39.213192 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:06:39.213117    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7\": container with ID starting with 35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7 not found: ID does not exist" containerID="35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7"
Sep 15 17:06:39.213192 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.213153    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7"} err="failed to get container status \"35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7\": rpc error: code = NotFound desc = could not find container \"35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7\": container with ID starting with 35f214bf4a30add3e4f9630d5f73d8ec230458a0f22ff50ff756a63c14ea8fc7 not found: ID does not exist"
Sep 15 17:06:39.213192 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.213179    2566 scope.go:117] "RemoveContainer" containerID="32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e"
Sep 15 17:06:39.213464 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:06:39.213448    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e\": container with ID starting with 32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e not found: ID does not exist" containerID="32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e"
Sep 15 17:06:39.213503 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.213470    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e"} err="failed to get container status \"32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e\": rpc error: code = NotFound desc = could not find container \"32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e\": container with ID starting with 32ca121a88c2bd96ad26c81ce132a0a294d3fbc975cf3cf81901b22d6de8f06e not found: ID does not exist"
Sep 15 17:06:39.213503 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.213486    2566 scope.go:117] "RemoveContainer" containerID="fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2"
Sep 15 17:06:39.213757 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:06:39.213738    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2\": container with ID starting with fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2 not found: ID does not exist" containerID="fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2"
Sep 15 17:06:39.213810 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.213764    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2"} err="failed to get container status \"fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2\": rpc error: code = NotFound desc = could not find container \"fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2\": container with ID starting with fd5a72ae1077966826e69b8192444281f3f7478fd9dcb655647cf6ae3ab5f2d2 not found: ID does not exist"
Sep 15 17:06:39.257733 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.257702    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b332cd3-f2b6-4349-ab85-7d5989a797d8-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:39.257733 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:39.257727    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rthmf\" (UniqueName: \"kubernetes.io/projected/9b332cd3-f2b6-4349-ab85-7d5989a797d8-kube-api-access-rthmf\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:40.159247 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:40.159212    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" path="/var/lib/kubelet/pods/9b332cd3-f2b6-4349-ab85-7d5989a797d8/volumes"
Sep 15 17:06:44.159390 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:44.159353    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:06:50.577420 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:50.577294    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"]
Sep 15 17:06:50.577912 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:50.577592    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" containerID="cri-o://c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e" gracePeriod=30
Sep 15 17:06:50.577912 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:50.577783    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kube-rbac-proxy" containerID="cri-o://124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a" gracePeriod=30
Sep 15 17:06:51.246633 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:51.246597    2566 generic.go:356] "Generic (PLEG): container finished" podID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerID="124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a" exitCode=2
Sep 15 17:06:51.246809 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:51.246596    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" event={"ID":"8c5a0bb5-d214-47b3-a1a7-278b09dde64e","Type":"ContainerDied","Data":"124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a"}
Sep 15 17:06:52.234890 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:52.234857    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:06:52.250367 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:52.250343    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:06:53.832992 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:53.832942    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.51:8643/healthz\": dial tcp 10.132.0.51:8643: connect: connection refused"
Sep 15 17:06:54.154814 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:54.154762    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.51:8080: connect: connection refused"
Sep 15 17:06:55.221879 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.221849    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:06:55.267480 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.267447    2566 generic.go:356] "Generic (PLEG): container finished" podID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerID="c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e" exitCode=0
Sep 15 17:06:55.267660 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.267498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" event={"ID":"8c5a0bb5-d214-47b3-a1a7-278b09dde64e","Type":"ContainerDied","Data":"c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e"}
Sep 15 17:06:55.267660 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.267528    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"
Sep 15 17:06:55.267660 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.267551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72" event={"ID":"8c5a0bb5-d214-47b3-a1a7-278b09dde64e","Type":"ContainerDied","Data":"472572b7b1e2175cd48a4ced703e65ff55c90295335bf1e8e7073f9b5d99fe66"}
Sep 15 17:06:55.267660 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.267600    2566 scope.go:117] "RemoveContainer" containerID="124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a"
Sep 15 17:06:55.277249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.277229    2566 scope.go:117] "RemoveContainer" containerID="c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e"
Sep 15 17:06:55.284670 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.284644    2566 scope.go:117] "RemoveContainer" containerID="c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4"
Sep 15 17:06:55.292085 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.292068    2566 scope.go:117] "RemoveContainer" containerID="124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a"
Sep 15 17:06:55.292337 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:06:55.292318    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a\": container with ID starting with 124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a not found: ID does not exist" containerID="124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a"
Sep 15 17:06:55.292410 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.292343    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a"} err="failed to get container status \"124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a\": rpc error: code = NotFound desc = could not find container \"124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a\": container with ID starting with 124e1d06159591ce07d9fac1b1c58543894db03d17536394761ba0478243eb7a not found: ID does not exist"
Sep 15 17:06:55.292410 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.292360    2566 scope.go:117] "RemoveContainer" containerID="c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e"
Sep 15 17:06:55.292628 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:06:55.292610    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e\": container with ID starting with c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e not found: ID does not exist" containerID="c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e"
Sep 15 17:06:55.292689 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.292634    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e"} err="failed to get container status \"c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e\": rpc error: code = NotFound desc = could not find container \"c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e\": container with ID starting with c4a142869b0a642a979f8a09370092cbb992a5e12c4272696c856426d076d18e not found: ID does not exist"
Sep 15 17:06:55.292689 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.292649    2566 scope.go:117] "RemoveContainer" containerID="c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4"
Sep 15 17:06:55.292911 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:06:55.292890    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4\": container with ID starting with c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4 not found: ID does not exist" containerID="c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4"
Sep 15 17:06:55.293002 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.292915    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4"} err="failed to get container status \"c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4\": rpc error: code = NotFound desc = could not find container \"c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4\": container with ID starting with c32378a9a544c1a9e3e928cc2c8b8f00517a21aa9592d4a1579a400a6b0625e4 not found: ID does not exist"
Sep 15 17:06:55.293002 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.292947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6tm5\" (UniqueName: \"kubernetes.io/projected/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-api-access-k6tm5\") pod \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") "
Sep 15 17:06:55.293002 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.292978    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-cabundle-cert\") pod \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") "
Sep 15 17:06:55.293152 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.293015    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls\") pod \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") "
Sep 15 17:06:55.293152 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.293043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kserve-provision-location\") pod \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") "
Sep 15 17:06:55.293152 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.293060    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-rbac-proxy-sar-config\") pod \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\" (UID: \"8c5a0bb5-d214-47b3-a1a7-278b09dde64e\") "
Sep 15 17:06:55.293396 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.293372    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8c5a0bb5-d214-47b3-a1a7-278b09dde64e" (UID: "8c5a0bb5-d214-47b3-a1a7-278b09dde64e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:06:55.293396 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.293385    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8c5a0bb5-d214-47b3-a1a7-278b09dde64e" (UID: "8c5a0bb5-d214-47b3-a1a7-278b09dde64e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:06:55.293914 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.293885    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c5a0bb5-d214-47b3-a1a7-278b09dde64e" (UID: "8c5a0bb5-d214-47b3-a1a7-278b09dde64e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:06:55.295076 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.295052    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-api-access-k6tm5" (OuterVolumeSpecName: "kube-api-access-k6tm5") pod "8c5a0bb5-d214-47b3-a1a7-278b09dde64e" (UID: "8c5a0bb5-d214-47b3-a1a7-278b09dde64e"). InnerVolumeSpecName "kube-api-access-k6tm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:06:55.295076 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.295071    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c5a0bb5-d214-47b3-a1a7-278b09dde64e" (UID: "8c5a0bb5-d214-47b3-a1a7-278b09dde64e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:06:55.393693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.393657    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6tm5\" (UniqueName: \"kubernetes.io/projected/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-api-access-k6tm5\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:55.393693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.393690    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:55.393889 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.393702    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:55.393889 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.393714    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:55.393889 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.393726    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c5a0bb5-d214-47b3-a1a7-278b09dde64e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:06:55.590637 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.590605    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"]
Sep 15 17:06:55.597201 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.597168    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-sbd72"]
Sep 15 17:06:55.771104 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771027    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"]
Sep 15 17:06:55.771378 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771363    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="storage-initializer"
Sep 15 17:06:55.771421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771380    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="storage-initializer"
Sep 15 17:06:55.771421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771390    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="storage-initializer"
Sep 15 17:06:55.771421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771396    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="storage-initializer"
Sep 15 17:06:55.771421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771404    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container"
Sep 15 17:06:55.771421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771411    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container"
Sep 15 17:06:55.771421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771418    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kube-rbac-proxy"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771428    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kube-rbac-proxy"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771438    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kube-rbac-proxy"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771444    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kube-rbac-proxy"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771453    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771458    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771531    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kube-rbac-proxy"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771539    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" containerName="kserve-container"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771548    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kube-rbac-proxy"
Sep 15 17:06:55.771611 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.771556    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b332cd3-f2b6-4349-ab85-7d5989a797d8" containerName="kserve-container"
Sep 15 17:06:55.775085 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.775065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.777727 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.777702    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:06:55.777890 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.777729    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:06:55.777890 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.777729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:06:55.777890 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.777711    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 15 17:06:55.777890 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.777765    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 15 17:06:55.777890 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.777756    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:06:55.778154 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.778139    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:06:55.786202 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.786178    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"]
Sep 15 17:06:55.796687 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.796657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.796820 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.796709    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.796820 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.796744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xslxb\" (UniqueName: \"kubernetes.io/projected/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-api-access-xslxb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.796889 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.796852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.796929 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.796912    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066e606e-2128-40f9-9518-8c1003c6dc2e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.897561 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.897515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066e606e-2128-40f9-9518-8c1003c6dc2e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.897805 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.897601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.897805 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.897625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.897805 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.897649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xslxb\" (UniqueName: \"kubernetes.io/projected/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-api-access-xslxb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.897805 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.897710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.898050 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:06:55.897838    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 15 17:06:55.898050 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:06:55.897900    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls podName:066e606e-2128-40f9-9518-8c1003c6dc2e nodeName:}" failed. No retries permitted until 2026-09-15 17:06:56.397879508 +0000 UTC m=+2404.765704691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" (UID: "066e606e-2128-40f9-9518-8c1003c6dc2e") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Sep 15 17:06:55.898050 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.898003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066e606e-2128-40f9-9518-8c1003c6dc2e-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.898323 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.898302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.898360 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.898325    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:55.905932 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:55.905911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xslxb\" (UniqueName: \"kubernetes.io/projected/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-api-access-xslxb\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:56.159180 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:56.159147    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c5a0bb5-d214-47b3-a1a7-278b09dde64e" path="/var/lib/kubelet/pods/8c5a0bb5-d214-47b3-a1a7-278b09dde64e/volumes"
Sep 15 17:06:56.402476 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:56.402437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:56.405058 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:56.405032    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:56.686168 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:56.686112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:06:56.961148 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:56.961117    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"]
Sep 15 17:06:56.963796 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:06:56.963769    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066e606e_2128_40f9_9518_8c1003c6dc2e.slice/crio-05bfd953cf7b3bbcaca939f31d2ba571fce1b464dbe2feea5f3e55fa179bf201 WatchSource:0}: Error finding container 05bfd953cf7b3bbcaca939f31d2ba571fce1b464dbe2feea5f3e55fa179bf201: Status 404 returned error can't find the container with id 05bfd953cf7b3bbcaca939f31d2ba571fce1b464dbe2feea5f3e55fa179bf201
Sep 15 17:06:57.278782 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:57.278680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" event={"ID":"066e606e-2128-40f9-9518-8c1003c6dc2e","Type":"ContainerStarted","Data":"a373040ee24f6efae8cdceb17490925eed5e9202e88f97267028eb580c4f0380"}
Sep 15 17:06:57.278782 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:06:57.278744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" event={"ID":"066e606e-2128-40f9-9518-8c1003c6dc2e","Type":"ContainerStarted","Data":"05bfd953cf7b3bbcaca939f31d2ba571fce1b464dbe2feea5f3e55fa179bf201"}
Sep 15 17:07:01.298020 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:01.297930    2566 generic.go:356] "Generic (PLEG): container finished" podID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerID="a373040ee24f6efae8cdceb17490925eed5e9202e88f97267028eb580c4f0380" exitCode=0
Sep 15 17:07:01.298020 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:01.297995    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" event={"ID":"066e606e-2128-40f9-9518-8c1003c6dc2e","Type":"ContainerDied","Data":"a373040ee24f6efae8cdceb17490925eed5e9202e88f97267028eb580c4f0380"}
Sep 15 17:07:02.304827 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:02.304792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" event={"ID":"066e606e-2128-40f9-9518-8c1003c6dc2e","Type":"ContainerStarted","Data":"ee30ac4e7a4fac99f3558882b583d8167775b476e4f59c87b402690f2774f133"}
Sep 15 17:07:02.305196 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:02.304852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" event={"ID":"066e606e-2128-40f9-9518-8c1003c6dc2e","Type":"ContainerStarted","Data":"82f49f2335fd19a35f46a070e705db93c0ea1962d37090b06f971f264465664e"}
Sep 15 17:07:02.305196 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:02.305107    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:07:02.305196 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:02.305151    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:07:02.306649 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:02.306626    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:07:02.325507 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:02.325462    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podStartSLOduration=3.384992853 podStartE2EDuration="7.325448056s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.940455203s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:07:02.322454869 +0000 UTC m=+2410.690280068" watchObservedRunningTime="2026-09-15 17:07:02.325448056 +0000 UTC m=+2410.693273324"
Sep 15 17:07:03.309086 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:03.309046    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:07:08.314159 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:08.314125    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:07:08.314668 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:08.314640    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:07:18.315346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:18.315256    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:07:28.315545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:28.315506    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:07:38.314828 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:38.314784    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:07:45.746253 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.746216    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"]
Sep 15 17:07:45.749104 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.749084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:45.751344 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.751310    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 15 17:07:45.751344 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.751329    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 15 17:07:45.759504 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.759477    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"]
Sep 15 17:07:45.928284 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.928252    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:45.928462 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.928307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:45.928462 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.928351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4422468-49b7-41cb-8286-6751c4746c94-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:45.928462 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.928379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:45.928659 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:45.928475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpb2\" (UniqueName: \"kubernetes.io/projected/a4422468-49b7-41cb-8286-6751c4746c94-kube-api-access-whpb2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.029963 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.029871    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.029963 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.029915    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4422468-49b7-41cb-8286-6751c4746c94-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.029963 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.029948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.030200 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.030001    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpb2\" (UniqueName: \"kubernetes.io/projected/a4422468-49b7-41cb-8286-6751c4746c94-kube-api-access-whpb2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.030200 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:07:46.030050    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 15 17:07:46.030200 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:07:46.030137    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls podName:a4422468-49b7-41cb-8286-6751c4746c94 nodeName:}" failed. No retries permitted until 2026-09-15 17:07:46.530116424 +0000 UTC m=+2454.897941605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-tmmjj" (UID: "a4422468-49b7-41cb-8286-6751c4746c94") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Sep 15 17:07:46.030200 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.030160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.030391 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.030348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4422468-49b7-41cb-8286-6751c4746c94-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.030841 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.030818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.030841 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.030833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.038217 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.038193    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpb2\" (UniqueName: \"kubernetes.io/projected/a4422468-49b7-41cb-8286-6751c4746c94-kube-api-access-whpb2\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.534726 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.534693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.537228 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.537194    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-tmmjj\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.660632 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.660561    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:46.930817 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:46.930790    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"]
Sep 15 17:07:46.932775 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:07:46.932741    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4422468_49b7_41cb_8286_6751c4746c94.slice/crio-72c1dd1632d187447ae189c957e1978715f2b0669af648b119c4fd596d2e7dee WatchSource:0}: Error finding container 72c1dd1632d187447ae189c957e1978715f2b0669af648b119c4fd596d2e7dee: Status 404 returned error can't find the container with id 72c1dd1632d187447ae189c957e1978715f2b0669af648b119c4fd596d2e7dee
Sep 15 17:07:47.514547 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:47.514509    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" event={"ID":"a4422468-49b7-41cb-8286-6751c4746c94","Type":"ContainerStarted","Data":"fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0"}
Sep 15 17:07:47.514724 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:47.514587    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" event={"ID":"a4422468-49b7-41cb-8286-6751c4746c94","Type":"ContainerStarted","Data":"72c1dd1632d187447ae189c957e1978715f2b0669af648b119c4fd596d2e7dee"}
Sep 15 17:07:48.314610 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:48.314549    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:07:51.533383 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:51.533344    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4422468-49b7-41cb-8286-6751c4746c94" containerID="fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0" exitCode=0
Sep 15 17:07:51.533851 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:51.533413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" event={"ID":"a4422468-49b7-41cb-8286-6751c4746c94","Type":"ContainerDied","Data":"fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0"}
Sep 15 17:07:52.540178 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:52.540139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" event={"ID":"a4422468-49b7-41cb-8286-6751c4746c94","Type":"ContainerStarted","Data":"01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e"}
Sep 15 17:07:52.540675 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:52.540215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" event={"ID":"a4422468-49b7-41cb-8286-6751c4746c94","Type":"ContainerStarted","Data":"d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b"}
Sep 15 17:07:52.540675 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:52.540504    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:52.540675 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:52.540554    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:52.542020 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:52.541995    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:07:52.559091 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:52.559037    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podStartSLOduration=3.767180803 podStartE2EDuration="7.559019457s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.791838654s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:07:52.558144272 +0000 UTC m=+2460.925969474" watchObservedRunningTime="2026-09-15 17:07:52.559019457 +0000 UTC m=+2460.926844657"
Sep 15 17:07:53.544748 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:53.544705    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:07:58.315325 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:58.315278    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:07:58.549342 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:58.549314    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:07:58.549835 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:07:58.549809    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:08:08.314845 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:08.314801    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:08:08.550377 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:08.550338    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:08:18.315672 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:18.315637    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:08:18.550584 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:18.550533    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:08:26.564803 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:26.564765    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"]
Sep 15 17:08:26.565217 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:26.565105    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" containerID="cri-o://82f49f2335fd19a35f46a070e705db93c0ea1962d37090b06f971f264465664e" gracePeriod=30
Sep 15 17:08:26.565321 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:26.565297    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kube-rbac-proxy" containerID="cri-o://ee30ac4e7a4fac99f3558882b583d8167775b476e4f59c87b402690f2774f133" gracePeriod=30
Sep 15 17:08:26.698343 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:26.698314    2566 generic.go:356] "Generic (PLEG): container finished" podID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerID="ee30ac4e7a4fac99f3558882b583d8167775b476e4f59c87b402690f2774f133" exitCode=2
Sep 15 17:08:26.698487 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:26.698376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" event={"ID":"066e606e-2128-40f9-9518-8c1003c6dc2e","Type":"ContainerDied","Data":"ee30ac4e7a4fac99f3558882b583d8167775b476e4f59c87b402690f2774f133"}
Sep 15 17:08:28.309674 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:28.309635    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.52:8643/healthz\": dial tcp 10.132.0.52:8643: connect: connection refused"
Sep 15 17:08:28.314977 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:28.314955    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.52:8080: connect: connection refused"
Sep 15 17:08:28.549793 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:28.549757    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:08:31.723790 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.723759    2566 generic.go:356] "Generic (PLEG): container finished" podID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerID="82f49f2335fd19a35f46a070e705db93c0ea1962d37090b06f971f264465664e" exitCode=0
Sep 15 17:08:31.724242 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.723830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" event={"ID":"066e606e-2128-40f9-9518-8c1003c6dc2e","Type":"ContainerDied","Data":"82f49f2335fd19a35f46a070e705db93c0ea1962d37090b06f971f264465664e"}
Sep 15 17:08:31.724242 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.723913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6" event={"ID":"066e606e-2128-40f9-9518-8c1003c6dc2e","Type":"ContainerDied","Data":"05bfd953cf7b3bbcaca939f31d2ba571fce1b464dbe2feea5f3e55fa179bf201"}
Sep 15 17:08:31.724242 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.723934    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05bfd953cf7b3bbcaca939f31d2ba571fce1b464dbe2feea5f3e55fa179bf201"
Sep 15 17:08:31.725661 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.725641    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:08:31.805557 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.805529    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-rbac-proxy-sar-config\") pod \"066e606e-2128-40f9-9518-8c1003c6dc2e\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") "
Sep 15 17:08:31.805742 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.805579    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-cabundle-cert\") pod \"066e606e-2128-40f9-9518-8c1003c6dc2e\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") "
Sep 15 17:08:31.805742 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.805608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls\") pod \"066e606e-2128-40f9-9518-8c1003c6dc2e\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") "
Sep 15 17:08:31.805742 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.805650    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xslxb\" (UniqueName: \"kubernetes.io/projected/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-api-access-xslxb\") pod \"066e606e-2128-40f9-9518-8c1003c6dc2e\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") "
Sep 15 17:08:31.805742 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.805670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066e606e-2128-40f9-9518-8c1003c6dc2e-kserve-provision-location\") pod \"066e606e-2128-40f9-9518-8c1003c6dc2e\" (UID: \"066e606e-2128-40f9-9518-8c1003c6dc2e\") "
Sep 15 17:08:31.806005 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.805978    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "066e606e-2128-40f9-9518-8c1003c6dc2e" (UID: "066e606e-2128-40f9-9518-8c1003c6dc2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:08:31.806167 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.806133    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "066e606e-2128-40f9-9518-8c1003c6dc2e" (UID: "066e606e-2128-40f9-9518-8c1003c6dc2e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:08:31.806274 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.806227    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066e606e-2128-40f9-9518-8c1003c6dc2e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "066e606e-2128-40f9-9518-8c1003c6dc2e" (UID: "066e606e-2128-40f9-9518-8c1003c6dc2e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:08:31.807882 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.807857    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "066e606e-2128-40f9-9518-8c1003c6dc2e" (UID: "066e606e-2128-40f9-9518-8c1003c6dc2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:08:31.808159 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.808134    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-api-access-xslxb" (OuterVolumeSpecName: "kube-api-access-xslxb") pod "066e606e-2128-40f9-9518-8c1003c6dc2e" (UID: "066e606e-2128-40f9-9518-8c1003c6dc2e"). InnerVolumeSpecName "kube-api-access-xslxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:08:31.907155 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.907118    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:08:31.907155 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.907151    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/066e606e-2128-40f9-9518-8c1003c6dc2e-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:08:31.907352 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.907163    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xslxb\" (UniqueName: \"kubernetes.io/projected/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-api-access-xslxb\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:08:31.907352 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.907178    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/066e606e-2128-40f9-9518-8c1003c6dc2e-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:08:31.907352 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:31.907190    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/066e606e-2128-40f9-9518-8c1003c6dc2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:08:32.729178 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:32.729147    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"
Sep 15 17:08:32.745903 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:32.745873    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"]
Sep 15 17:08:32.751854 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:32.751827    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-nmlv6"]
Sep 15 17:08:34.159540 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:34.159504    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" path="/var/lib/kubelet/pods/066e606e-2128-40f9-9518-8c1003c6dc2e/volumes"
Sep 15 17:08:38.550745 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:38.550705    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:08:48.550545 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:48.550452    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:08:58.551425 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:08:58.551386    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:09:05.829752 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:05.829716    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"]
Sep 15 17:09:05.830161 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:05.830057    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" containerID="cri-o://d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b" gracePeriod=30
Sep 15 17:09:05.830161 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:05.830077    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kube-rbac-proxy" containerID="cri-o://01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e" gracePeriod=30
Sep 15 17:09:06.897420 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:06.897380    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4422468-49b7-41cb-8286-6751c4746c94" containerID="01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e" exitCode=2
Sep 15 17:09:06.897808 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:06.897450    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" event={"ID":"a4422468-49b7-41cb-8286-6751c4746c94","Type":"ContainerDied","Data":"01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e"}
Sep 15 17:09:08.545601 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:08.545542    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.53:8643/healthz\": dial tcp 10.132.0.53:8643: connect: connection refused"
Sep 15 17:09:08.549893 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:08.549870    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.53:8080: connect: connection refused"
Sep 15 17:09:09.575102 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.575076    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:09:09.630070 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.629996    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls\") pod \"a4422468-49b7-41cb-8286-6751c4746c94\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") "
Sep 15 17:09:09.630070 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.630036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-kube-rbac-proxy-sar-config\") pod \"a4422468-49b7-41cb-8286-6751c4746c94\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") "
Sep 15 17:09:09.630307 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.630080    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4422468-49b7-41cb-8286-6751c4746c94-kserve-provision-location\") pod \"a4422468-49b7-41cb-8286-6751c4746c94\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") "
Sep 15 17:09:09.630307 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.630185    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whpb2\" (UniqueName: \"kubernetes.io/projected/a4422468-49b7-41cb-8286-6751c4746c94-kube-api-access-whpb2\") pod \"a4422468-49b7-41cb-8286-6751c4746c94\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") "
Sep 15 17:09:09.630307 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.630227    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-cabundle-cert\") pod \"a4422468-49b7-41cb-8286-6751c4746c94\" (UID: \"a4422468-49b7-41cb-8286-6751c4746c94\") "
Sep 15 17:09:09.630486 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.630367    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4422468-49b7-41cb-8286-6751c4746c94-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a4422468-49b7-41cb-8286-6751c4746c94" (UID: "a4422468-49b7-41cb-8286-6751c4746c94"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:09:09.630544 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.630509    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a4422468-49b7-41cb-8286-6751c4746c94-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:09.630627 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.630545    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a4422468-49b7-41cb-8286-6751c4746c94" (UID: "a4422468-49b7-41cb-8286-6751c4746c94"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:09:09.630627 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.630599    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a4422468-49b7-41cb-8286-6751c4746c94" (UID: "a4422468-49b7-41cb-8286-6751c4746c94"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:09:09.632237 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.632209    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4422468-49b7-41cb-8286-6751c4746c94-kube-api-access-whpb2" (OuterVolumeSpecName: "kube-api-access-whpb2") pod "a4422468-49b7-41cb-8286-6751c4746c94" (UID: "a4422468-49b7-41cb-8286-6751c4746c94"). InnerVolumeSpecName "kube-api-access-whpb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:09:09.632237 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.632224    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a4422468-49b7-41cb-8286-6751c4746c94" (UID: "a4422468-49b7-41cb-8286-6751c4746c94"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:09:09.731829 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.731794    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whpb2\" (UniqueName: \"kubernetes.io/projected/a4422468-49b7-41cb-8286-6751c4746c94-kube-api-access-whpb2\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:09.731829 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.731822    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:09.731829 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.731831    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a4422468-49b7-41cb-8286-6751c4746c94-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:09.732055 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.731840    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a4422468-49b7-41cb-8286-6751c4746c94-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:09.912184 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.912145    2566 generic.go:356] "Generic (PLEG): container finished" podID="a4422468-49b7-41cb-8286-6751c4746c94" containerID="d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b" exitCode=0
Sep 15 17:09:09.912373 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.912210    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"
Sep 15 17:09:09.912373 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.912231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" event={"ID":"a4422468-49b7-41cb-8286-6751c4746c94","Type":"ContainerDied","Data":"d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b"}
Sep 15 17:09:09.912373 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.912294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj" event={"ID":"a4422468-49b7-41cb-8286-6751c4746c94","Type":"ContainerDied","Data":"72c1dd1632d187447ae189c957e1978715f2b0669af648b119c4fd596d2e7dee"}
Sep 15 17:09:09.912373 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.912321    2566 scope.go:117] "RemoveContainer" containerID="01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e"
Sep 15 17:09:09.921253 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.921238    2566 scope.go:117] "RemoveContainer" containerID="d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b"
Sep 15 17:09:09.928741 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.928717    2566 scope.go:117] "RemoveContainer" containerID="fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0"
Sep 15 17:09:09.936757 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.936728    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"]
Sep 15 17:09:09.939882 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.939857    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-tmmjj"]
Sep 15 17:09:09.942317 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.942299    2566 scope.go:117] "RemoveContainer" containerID="01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e"
Sep 15 17:09:09.942635 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:09:09.942615    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e\": container with ID starting with 01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e not found: ID does not exist" containerID="01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e"
Sep 15 17:09:09.942693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.942643    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e"} err="failed to get container status \"01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e\": rpc error: code = NotFound desc = could not find container \"01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e\": container with ID starting with 01998cb6f30f21642e7c321a6936d6072bc9283b927d6733e7437b92fa9e463e not found: ID does not exist"
Sep 15 17:09:09.942693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.942662    2566 scope.go:117] "RemoveContainer" containerID="d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b"
Sep 15 17:09:09.942920 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:09:09.942902    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b\": container with ID starting with d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b not found: ID does not exist" containerID="d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b"
Sep 15 17:09:09.942992 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.942929    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b"} err="failed to get container status \"d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b\": rpc error: code = NotFound desc = could not find container \"d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b\": container with ID starting with d9ee7620937eeeae7e05e463bac9ee2091025bb47b25a9ba5e31cb0fc11e426b not found: ID does not exist"
Sep 15 17:09:09.942992 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.942949    2566 scope.go:117] "RemoveContainer" containerID="fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0"
Sep 15 17:09:09.943214 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:09:09.943195    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0\": container with ID starting with fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0 not found: ID does not exist" containerID="fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0"
Sep 15 17:09:09.943254 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:09.943220    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0"} err="failed to get container status \"fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0\": rpc error: code = NotFound desc = could not find container \"fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0\": container with ID starting with fbb1efdbb7d7c3d031de36c80c95160db4c998e84b4305ef8b4e66a2afbc9fd0 not found: ID does not exist"
Sep 15 17:09:10.159001 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:10.158969    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4422468-49b7-41cb-8286-6751c4746c94" path="/var/lib/kubelet/pods/a4422468-49b7-41cb-8286-6751c4746c94/volumes"
Sep 15 17:09:11.033199 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033168    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"]
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033490    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kube-rbac-proxy"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033500    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kube-rbac-proxy"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033510    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="storage-initializer"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033517    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="storage-initializer"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033525    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kube-rbac-proxy"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033531    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kube-rbac-proxy"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033541    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033547    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033554    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033559    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033586    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="storage-initializer"
Sep 15 17:09:11.033593 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033594    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="storage-initializer"
Sep 15 17:09:11.033960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033643    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kube-rbac-proxy"
Sep 15 17:09:11.033960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033651    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kube-rbac-proxy"
Sep 15 17:09:11.033960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033657    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4422468-49b7-41cb-8286-6751c4746c94" containerName="kserve-container"
Sep 15 17:09:11.033960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.033664    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="066e606e-2128-40f9-9518-8c1003c6dc2e" containerName="kserve-container"
Sep 15 17:09:11.038287 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.038267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.040878 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.040851    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Sep 15 17:09:11.041002 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.040877    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:09:11.041217 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.041199    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 15 17:09:11.041330 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.041221    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:09:11.041330 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.041305    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Sep 15 17:09:11.041449 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.041380    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:09:11.046465 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.046442    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"]
Sep 15 17:09:11.144030 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.143995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d644643f-f26f-4359-8b19-ffdad8a33cb4-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.144209 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.144038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.144209 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.144086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d644643f-f26f-4359-8b19-ffdad8a33cb4-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.144209 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.144125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8tc6\" (UniqueName: \"kubernetes.io/projected/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-api-access-q8tc6\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.245451 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.245413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d644643f-f26f-4359-8b19-ffdad8a33cb4-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.245649 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.245454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.245649 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.245477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d644643f-f26f-4359-8b19-ffdad8a33cb4-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.245649 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.245505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8tc6\" (UniqueName: \"kubernetes.io/projected/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-api-access-q8tc6\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.245903 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.245884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d644643f-f26f-4359-8b19-ffdad8a33cb4-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.246175 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.246154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.247959 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.247932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d644643f-f26f-4359-8b19-ffdad8a33cb4-proxy-tls\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.253973 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.253948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8tc6\" (UniqueName: \"kubernetes.io/projected/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-api-access-q8tc6\") pod \"isvc-sklearn-s3-predictor-6589c9c8f6-nt57c\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.349651 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.349536    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:11.620290 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.620263    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"]
Sep 15 17:09:11.622776 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:09:11.622747    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd644643f_f26f_4359_8b19_ffdad8a33cb4.slice/crio-fc1cd3f405dfad7af62582abee81cdf584e20ea8e8e97e27bcd246c8dfdbed1b WatchSource:0}: Error finding container fc1cd3f405dfad7af62582abee81cdf584e20ea8e8e97e27bcd246c8dfdbed1b: Status 404 returned error can't find the container with id fc1cd3f405dfad7af62582abee81cdf584e20ea8e8e97e27bcd246c8dfdbed1b
Sep 15 17:09:11.924956 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.924915    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" event={"ID":"d644643f-f26f-4359-8b19-ffdad8a33cb4","Type":"ContainerStarted","Data":"a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0"}
Sep 15 17:09:11.925129 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:11.925001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" event={"ID":"d644643f-f26f-4359-8b19-ffdad8a33cb4","Type":"ContainerStarted","Data":"fc1cd3f405dfad7af62582abee81cdf584e20ea8e8e97e27bcd246c8dfdbed1b"}
Sep 15 17:09:12.930374 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:12.930333    2566 generic.go:356] "Generic (PLEG): container finished" podID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerID="a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0" exitCode=0
Sep 15 17:09:12.930905 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:12.930393    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" event={"ID":"d644643f-f26f-4359-8b19-ffdad8a33cb4","Type":"ContainerDied","Data":"a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0"}
Sep 15 17:09:13.937632 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:13.937592    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" event={"ID":"d644643f-f26f-4359-8b19-ffdad8a33cb4","Type":"ContainerStarted","Data":"ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c"}
Sep 15 17:09:13.938077 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:13.937659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" event={"ID":"d644643f-f26f-4359-8b19-ffdad8a33cb4","Type":"ContainerStarted","Data":"3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d"}
Sep 15 17:09:13.938077 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:13.937718    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:13.955886 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:13.955827    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podStartSLOduration=1.8421080380000001 podStartE2EDuration="2.955807721s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.113699683s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:09:13.954091525 +0000 UTC m=+2542.321916726" watchObservedRunningTime="2026-09-15 17:09:13.955807721 +0000 UTC m=+2542.323632921"
Sep 15 17:09:14.942865 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:14.942827    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:14.944131 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:14.944110    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:09:15.947293 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:15.947243    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:09:20.951826 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:20.951793    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:09:20.952411 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:20.952385    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:09:27.087015 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.086979    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"]
Sep 15 17:09:27.090583 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.090545    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.092899 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.092875    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 15 17:09:27.092899 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.092889    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:09:27.093098 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.092974    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 15 17:09:27.093098 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.092975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:09:27.099298 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.099275    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"]
Sep 15 17:09:27.179489 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.179449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9396b8-0227-4429-a043-b418b7c6eea6-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.179679 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.179499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.179679 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.179517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4899\" (UniqueName: \"kubernetes.io/projected/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-api-access-g4899\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.179679 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.179606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.179679 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.179660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9396b8-0227-4429-a043-b418b7c6eea6-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.281087 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.281052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9396b8-0227-4429-a043-b418b7c6eea6-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.281251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.281098    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.281251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.281123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4899\" (UniqueName: \"kubernetes.io/projected/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-api-access-g4899\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.281251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.281151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.281251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.281189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9396b8-0227-4429-a043-b418b7c6eea6-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.281652 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.281631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9396b8-0227-4429-a043-b418b7c6eea6-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.281813 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.281792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.281858 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.281830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.283625 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.283606    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9396b8-0227-4429-a043-b418b7c6eea6-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.288143 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.288119    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4899\" (UniqueName: \"kubernetes.io/projected/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-api-access-g4899\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.401989 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.401960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:27.673540 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:27.673509    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"]
Sep 15 17:09:27.675595 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:09:27.675543    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9396b8_0227_4429_a043_b418b7c6eea6.slice/crio-d6d91253ec2b93fd8e5b892f646ef0c875f5e719ab08c2f82399797dcf38ba85 WatchSource:0}: Error finding container d6d91253ec2b93fd8e5b892f646ef0c875f5e719ab08c2f82399797dcf38ba85: Status 404 returned error can't find the container with id d6d91253ec2b93fd8e5b892f646ef0c875f5e719ab08c2f82399797dcf38ba85
Sep 15 17:09:28.004148 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:28.004039    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" event={"ID":"9c9396b8-0227-4429-a043-b418b7c6eea6","Type":"ContainerStarted","Data":"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca"}
Sep 15 17:09:28.004148 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:28.004120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" event={"ID":"9c9396b8-0227-4429-a043-b418b7c6eea6","Type":"ContainerStarted","Data":"d6d91253ec2b93fd8e5b892f646ef0c875f5e719ab08c2f82399797dcf38ba85"}
Sep 15 17:09:30.952465 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:30.952424    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:09:32.025762 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:32.025726    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerID="122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca" exitCode=0
Sep 15 17:09:32.026172 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:32.025774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" event={"ID":"9c9396b8-0227-4429-a043-b418b7c6eea6","Type":"ContainerDied","Data":"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca"}
Sep 15 17:09:33.034263 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:33.034223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" event={"ID":"9c9396b8-0227-4429-a043-b418b7c6eea6","Type":"ContainerStarted","Data":"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e"}
Sep 15 17:09:33.034718 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:33.034289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" event={"ID":"9c9396b8-0227-4429-a043-b418b7c6eea6","Type":"ContainerStarted","Data":"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba"}
Sep 15 17:09:33.034718 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:33.034493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:33.053174 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:33.053126    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" podStartSLOduration=1.9972560499999998 podStartE2EDuration="6.053109798s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.055853748s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:09:33.052245151 +0000 UTC m=+2561.420070377" watchObservedRunningTime="2026-09-15 17:09:33.053109798 +0000 UTC m=+2561.420935074"
Sep 15 17:09:34.040304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:34.040260    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:34.041818 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:34.041789    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 15 17:09:35.052054 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:35.052015    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.55:8080: connect: connection refused"
Sep 15 17:09:40.057364 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:40.057335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:40.058713 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:40.058694    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:40.952449 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:40.952410    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:09:47.146128 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.146088    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"]
Sep 15 17:09:47.146605 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.146413    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kserve-container" containerID="cri-o://bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba" gracePeriod=30
Sep 15 17:09:47.146605 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.146460    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kube-rbac-proxy" containerID="cri-o://734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e" gracePeriod=30
Sep 15 17:09:47.793120 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.793093    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:47.851315 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.851228    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-cabundle-cert\") pod \"9c9396b8-0227-4429-a043-b418b7c6eea6\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") "
Sep 15 17:09:47.851315 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.851268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9396b8-0227-4429-a043-b418b7c6eea6-kserve-provision-location\") pod \"9c9396b8-0227-4429-a043-b418b7c6eea6\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") "
Sep 15 17:09:47.851315 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.851305    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9396b8-0227-4429-a043-b418b7c6eea6-proxy-tls\") pod \"9c9396b8-0227-4429-a043-b418b7c6eea6\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") "
Sep 15 17:09:47.851616 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.851324    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4899\" (UniqueName: \"kubernetes.io/projected/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-api-access-g4899\") pod \"9c9396b8-0227-4429-a043-b418b7c6eea6\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") "
Sep 15 17:09:47.851616 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.851340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-rbac-proxy-sar-config\") pod \"9c9396b8-0227-4429-a043-b418b7c6eea6\" (UID: \"9c9396b8-0227-4429-a043-b418b7c6eea6\") "
Sep 15 17:09:47.851811 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.851776    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9c9396b8-0227-4429-a043-b418b7c6eea6" (UID: "9c9396b8-0227-4429-a043-b418b7c6eea6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:09:47.851811 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.851798    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9396b8-0227-4429-a043-b418b7c6eea6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9c9396b8-0227-4429-a043-b418b7c6eea6" (UID: "9c9396b8-0227-4429-a043-b418b7c6eea6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:09:47.851994 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.851819    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9c9396b8-0227-4429-a043-b418b7c6eea6" (UID: "9c9396b8-0227-4429-a043-b418b7c6eea6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:09:47.853510 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.853489    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c9396b8-0227-4429-a043-b418b7c6eea6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9c9396b8-0227-4429-a043-b418b7c6eea6" (UID: "9c9396b8-0227-4429-a043-b418b7c6eea6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:09:47.853633 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.853512    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-api-access-g4899" (OuterVolumeSpecName: "kube-api-access-g4899") pod "9c9396b8-0227-4429-a043-b418b7c6eea6" (UID: "9c9396b8-0227-4429-a043-b418b7c6eea6"). InnerVolumeSpecName "kube-api-access-g4899". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:09:47.951986 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.951952    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:47.951986 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.951982    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9c9396b8-0227-4429-a043-b418b7c6eea6-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:47.951986 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.951993    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9c9396b8-0227-4429-a043-b418b7c6eea6-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:47.952186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.952002    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4899\" (UniqueName: \"kubernetes.io/projected/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-api-access-g4899\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:47.952186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:47.952010    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9c9396b8-0227-4429-a043-b418b7c6eea6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:09:48.117824 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.117733    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerID="734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e" exitCode=2
Sep 15 17:09:48.117824 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.117758    2566 generic.go:356] "Generic (PLEG): container finished" podID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerID="bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba" exitCode=0
Sep 15 17:09:48.117824 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.117799    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"
Sep 15 17:09:48.117824 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.117815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" event={"ID":"9c9396b8-0227-4429-a043-b418b7c6eea6","Type":"ContainerDied","Data":"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e"}
Sep 15 17:09:48.118146 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.117880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" event={"ID":"9c9396b8-0227-4429-a043-b418b7c6eea6","Type":"ContainerDied","Data":"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba"}
Sep 15 17:09:48.118146 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.117905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df" event={"ID":"9c9396b8-0227-4429-a043-b418b7c6eea6","Type":"ContainerDied","Data":"d6d91253ec2b93fd8e5b892f646ef0c875f5e719ab08c2f82399797dcf38ba85"}
Sep 15 17:09:48.118146 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.117931    2566 scope.go:117] "RemoveContainer" containerID="734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e"
Sep 15 17:09:48.126609 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.126591    2566 scope.go:117] "RemoveContainer" containerID="bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba"
Sep 15 17:09:48.136671 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.135246    2566 scope.go:117] "RemoveContainer" containerID="122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca"
Sep 15 17:09:48.141525 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.141505    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"]
Sep 15 17:09:48.145376 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.145354    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-gg9df"]
Sep 15 17:09:48.145886 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.145869    2566 scope.go:117] "RemoveContainer" containerID="734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e"
Sep 15 17:09:48.146156 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:09:48.146137    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e\": container with ID starting with 734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e not found: ID does not exist" containerID="734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e"
Sep 15 17:09:48.146432 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.146163    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e"} err="failed to get container status \"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e\": rpc error: code = NotFound desc = could not find container \"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e\": container with ID starting with 734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e not found: ID does not exist"
Sep 15 17:09:48.146432 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.146180    2566 scope.go:117] "RemoveContainer" containerID="bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba"
Sep 15 17:09:48.146432 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:09:48.146400    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba\": container with ID starting with bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba not found: ID does not exist" containerID="bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba"
Sep 15 17:09:48.146579 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.146427    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba"} err="failed to get container status \"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba\": rpc error: code = NotFound desc = could not find container \"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba\": container with ID starting with bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba not found: ID does not exist"
Sep 15 17:09:48.146579 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.146445    2566 scope.go:117] "RemoveContainer" containerID="122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca"
Sep 15 17:09:48.146693 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:09:48.146679    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca\": container with ID starting with 122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca not found: ID does not exist" containerID="122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca"
Sep 15 17:09:48.146739 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.146696    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca"} err="failed to get container status \"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca\": rpc error: code = NotFound desc = could not find container \"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca\": container with ID starting with 122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca not found: ID does not exist"
Sep 15 17:09:48.146739 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.146707    2566 scope.go:117] "RemoveContainer" containerID="734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e"
Sep 15 17:09:48.146926 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.146910    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e"} err="failed to get container status \"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e\": rpc error: code = NotFound desc = could not find container \"734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e\": container with ID starting with 734029ab2302fc8a3ff9263feabb43616a7f1723f619d5ab02a6748f247fcb4e not found: ID does not exist"
Sep 15 17:09:48.146926 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.146926    2566 scope.go:117] "RemoveContainer" containerID="bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba"
Sep 15 17:09:48.147134 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.147115    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba"} err="failed to get container status \"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba\": rpc error: code = NotFound desc = could not find container \"bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba\": container with ID starting with bfad4e5ed0c9e263c80805969ec914ec68173f78c642ba5bcedbd845b0b3e9ba not found: ID does not exist"
Sep 15 17:09:48.147186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.147136    2566 scope.go:117] "RemoveContainer" containerID="122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca"
Sep 15 17:09:48.147345 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.147326    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca"} err="failed to get container status \"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca\": rpc error: code = NotFound desc = could not find container \"122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca\": container with ID starting with 122a2691c6ef0427bc4bcbe63d52a146b6b214674354c0831fda9a3895b012ca not found: ID does not exist"
Sep 15 17:09:48.159586 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:48.159546    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" path="/var/lib/kubelet/pods/9c9396b8-0227-4429-a043-b418b7c6eea6/volumes"
Sep 15 17:09:50.952760 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:09:50.952720    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:10:00.952720 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:00.952678    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:10:10.952784 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:10.952743    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:10:20.952457 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:20.952369    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:10:22.156195 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:22.156154    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:10:32.158798 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:32.158768    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:10:41.115082 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:41.115040    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"]
Sep 15 17:10:41.115532 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:41.115278    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" containerID="cri-o://3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d" gracePeriod=30
Sep 15 17:10:41.115532 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:41.115300    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kube-rbac-proxy" containerID="cri-o://ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c" gracePeriod=30
Sep 15 17:10:41.361870 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:41.361831    2566 generic.go:356] "Generic (PLEG): container finished" podID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerID="ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c" exitCode=2
Sep 15 17:10:41.361870 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:41.361874    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" event={"ID":"d644643f-f26f-4359-8b19-ffdad8a33cb4","Type":"ContainerDied","Data":"ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c"}
Sep 15 17:10:42.156711 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:42.156672    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.54:8080: connect: connection refused"
Sep 15 17:10:45.968505 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:45.968478    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:10:46.132461 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.132373    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d644643f-f26f-4359-8b19-ffdad8a33cb4-proxy-tls\") pod \"d644643f-f26f-4359-8b19-ffdad8a33cb4\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") "
Sep 15 17:10:46.132461 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.132409    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d644643f-f26f-4359-8b19-ffdad8a33cb4-kserve-provision-location\") pod \"d644643f-f26f-4359-8b19-ffdad8a33cb4\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") "
Sep 15 17:10:46.132461 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.132434    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8tc6\" (UniqueName: \"kubernetes.io/projected/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-api-access-q8tc6\") pod \"d644643f-f26f-4359-8b19-ffdad8a33cb4\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") "
Sep 15 17:10:46.132461 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.132456    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-rbac-proxy-sar-config\") pod \"d644643f-f26f-4359-8b19-ffdad8a33cb4\" (UID: \"d644643f-f26f-4359-8b19-ffdad8a33cb4\") "
Sep 15 17:10:46.132842 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.132812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d644643f-f26f-4359-8b19-ffdad8a33cb4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d644643f-f26f-4359-8b19-ffdad8a33cb4" (UID: "d644643f-f26f-4359-8b19-ffdad8a33cb4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:10:46.132902 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.132848    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d644643f-f26f-4359-8b19-ffdad8a33cb4" (UID: "d644643f-f26f-4359-8b19-ffdad8a33cb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:10:46.134581 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.134519    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d644643f-f26f-4359-8b19-ffdad8a33cb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d644643f-f26f-4359-8b19-ffdad8a33cb4" (UID: "d644643f-f26f-4359-8b19-ffdad8a33cb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:10:46.134695 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.134629    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-api-access-q8tc6" (OuterVolumeSpecName: "kube-api-access-q8tc6") pod "d644643f-f26f-4359-8b19-ffdad8a33cb4" (UID: "d644643f-f26f-4359-8b19-ffdad8a33cb4"). InnerVolumeSpecName "kube-api-access-q8tc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:10:46.233737 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.233708    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d644643f-f26f-4359-8b19-ffdad8a33cb4-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:10:46.233737 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.233735    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d644643f-f26f-4359-8b19-ffdad8a33cb4-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:10:46.233737 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.233744    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8tc6\" (UniqueName: \"kubernetes.io/projected/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-api-access-q8tc6\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:10:46.233943 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.233753    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d644643f-f26f-4359-8b19-ffdad8a33cb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:10:46.388518 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.388428    2566 generic.go:356] "Generic (PLEG): container finished" podID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerID="3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d" exitCode=0
Sep 15 17:10:46.388518 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.388486    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"
Sep 15 17:10:46.388734 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.388514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" event={"ID":"d644643f-f26f-4359-8b19-ffdad8a33cb4","Type":"ContainerDied","Data":"3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d"}
Sep 15 17:10:46.388734 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.388589    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" event={"ID":"d644643f-f26f-4359-8b19-ffdad8a33cb4","Type":"ContainerDied","Data":"fc1cd3f405dfad7af62582abee81cdf584e20ea8e8e97e27bcd246c8dfdbed1b"}
Sep 15 17:10:46.388734 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.388617    2566 scope.go:117] "RemoveContainer" containerID="ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c"
Sep 15 17:10:46.396784 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.396763    2566 scope.go:117] "RemoveContainer" containerID="3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d"
Sep 15 17:10:46.404474 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.404458    2566 scope.go:117] "RemoveContainer" containerID="a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0"
Sep 15 17:10:46.408079 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.408059    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"]
Sep 15 17:10:46.412357 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.412335    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c"]
Sep 15 17:10:46.412996 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.412981    2566 scope.go:117] "RemoveContainer" containerID="ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c"
Sep 15 17:10:46.413259 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:10:46.413243    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c\": container with ID starting with ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c not found: ID does not exist" containerID="ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c"
Sep 15 17:10:46.413304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.413267    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c"} err="failed to get container status \"ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c\": rpc error: code = NotFound desc = could not find container \"ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c\": container with ID starting with ed94427eafe5b7bf6f4ad23c58e55e3e7c42c4f7b0fa189c82aad57dd7beb85c not found: ID does not exist"
Sep 15 17:10:46.413304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.413283    2566 scope.go:117] "RemoveContainer" containerID="3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d"
Sep 15 17:10:46.413514 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:10:46.413500    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d\": container with ID starting with 3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d not found: ID does not exist" containerID="3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d"
Sep 15 17:10:46.413556 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.413518    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d"} err="failed to get container status \"3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d\": rpc error: code = NotFound desc = could not find container \"3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d\": container with ID starting with 3a8c8b83be4b8ee47de8b2214901e4c72434f822f137912d7f424fc982a6be4d not found: ID does not exist"
Sep 15 17:10:46.413556 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.413531    2566 scope.go:117] "RemoveContainer" containerID="a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0"
Sep 15 17:10:46.413765 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:10:46.413748    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0\": container with ID starting with a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0 not found: ID does not exist" containerID="a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0"
Sep 15 17:10:46.413810 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.413770    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0"} err="failed to get container status \"a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0\": rpc error: code = NotFound desc = could not find container \"a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0\": container with ID starting with a2d1642bb1416d30bdc412ee958a4683d5b2fb14fb07dd406e5fe10e91d277c0 not found: ID does not exist"
Sep 15 17:10:46.949287 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:46.949246    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-6589c9c8f6-nt57c" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.54:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 15 17:10:47.559854 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.559818    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"]
Sep 15 17:10:47.560249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560188    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kube-rbac-proxy"
Sep 15 17:10:47.560249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560201    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kube-rbac-proxy"
Sep 15 17:10:47.560249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560211    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container"
Sep 15 17:10:47.560249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560217    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container"
Sep 15 17:10:47.560249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560223    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kserve-container"
Sep 15 17:10:47.560249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560230    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kserve-container"
Sep 15 17:10:47.560249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560244    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="storage-initializer"
Sep 15 17:10:47.560249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560251    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="storage-initializer"
Sep 15 17:10:47.560529 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560267    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="storage-initializer"
Sep 15 17:10:47.560529 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560274    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="storage-initializer"
Sep 15 17:10:47.560529 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560283    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kube-rbac-proxy"
Sep 15 17:10:47.560529 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560288    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kube-rbac-proxy"
Sep 15 17:10:47.560529 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560350    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kube-rbac-proxy"
Sep 15 17:10:47.560529 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560358    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kube-rbac-proxy"
Sep 15 17:10:47.560529 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560364    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9396b8-0227-4429-a043-b418b7c6eea6" containerName="kserve-container"
Sep 15 17:10:47.560529 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.560373    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" containerName="kserve-container"
Sep 15 17:10:47.565078 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.565051    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.567472 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.567441    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:10:47.567646 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.567631    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 15 17:10:47.567968 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.567793    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:10:47.567968 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.567819    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:10:47.567968 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.567793    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:10:47.568502 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.568485    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 15 17:10:47.568616 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.568536    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:10:47.576355 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.576331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"]
Sep 15 17:10:47.745344 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.745308    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54lg6\" (UniqueName: \"kubernetes.io/projected/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-api-access-54lg6\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.745344 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.745343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.745555 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.745364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/539d2b25-44f0-4d09-8d67-96f87b0b309e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.745555 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.745389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.745555 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.745499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.846378 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.846284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.846378 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.846372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54lg6\" (UniqueName: \"kubernetes.io/projected/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-api-access-54lg6\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.846631 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.846402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.846631 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.846423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/539d2b25-44f0-4d09-8d67-96f87b0b309e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.846631 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.846445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.846631 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:10:47.846565    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 15 17:10:47.846797 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:10:47.846680    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls podName:539d2b25-44f0-4d09-8d67-96f87b0b309e nodeName:}" failed. No retries permitted until 2026-09-15 17:10:48.346659721 +0000 UTC m=+2636.714484906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls") pod "isvc-sklearn-mcp-predictor-56686cd547-9jbr5" (UID: "539d2b25-44f0-4d09-8d67-96f87b0b309e") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 15 17:10:47.846871 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.846844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/539d2b25-44f0-4d09-8d67-96f87b0b309e-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.847122 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.847103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.847165 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.847104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:47.856758 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:47.856730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54lg6\" (UniqueName: \"kubernetes.io/projected/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-api-access-54lg6\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:48.159069 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:48.159036    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d644643f-f26f-4359-8b19-ffdad8a33cb4" path="/var/lib/kubelet/pods/d644643f-f26f-4359-8b19-ffdad8a33cb4/volumes"
Sep 15 17:10:48.351410 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:48.351373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:48.353875 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:48.353855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-56686cd547-9jbr5\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:48.476226 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:48.476142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:48.750842 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:48.750763    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"]
Sep 15 17:10:48.753823 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:10:48.753782    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539d2b25_44f0_4d09_8d67_96f87b0b309e.slice/crio-a78f9cab3d43367604c2d2c6e5a661bce63466334e28ecd1ea77a43cd4a2e812 WatchSource:0}: Error finding container a78f9cab3d43367604c2d2c6e5a661bce63466334e28ecd1ea77a43cd4a2e812: Status 404 returned error can't find the container with id a78f9cab3d43367604c2d2c6e5a661bce63466334e28ecd1ea77a43cd4a2e812
Sep 15 17:10:49.406646 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:49.406609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerStarted","Data":"9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896"}
Sep 15 17:10:49.406815 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:49.406674    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerStarted","Data":"a78f9cab3d43367604c2d2c6e5a661bce63466334e28ecd1ea77a43cd4a2e812"}
Sep 15 17:10:53.425938 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:53.425902    2566 generic.go:356] "Generic (PLEG): container finished" podID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerID="9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896" exitCode=0
Sep 15 17:10:53.426338 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:53.425975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerDied","Data":"9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896"}
Sep 15 17:10:53.484185 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:53.484167    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 15 17:10:54.436286 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:54.436248    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerStarted","Data":"b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b"}
Sep 15 17:10:57.453395 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:57.453359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerStarted","Data":"684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4"}
Sep 15 17:10:57.453835 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:57.453421    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerStarted","Data":"6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38"}
Sep 15 17:10:57.453835 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:57.453591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:57.453835 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:57.453623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:57.453835 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:57.453645    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:10:57.475911 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:10:57.475865    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podStartSLOduration=3.490587105 podStartE2EDuration="10.475849107s" totalImagesPullingTime="2.939228017s" totalInitContainerRuntime="4.046033985s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:10:57.475732886 +0000 UTC m=+2645.843558085" watchObservedRunningTime="2026-09-15 17:10:57.475849107 +0000 UTC m=+2645.843674326"
Sep 15 17:11:03.463662 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:11:03.463625    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:11:23.465329 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:11:23.465285    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 15 17:11:28.476701 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:11:28.476660    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 15 17:11:33.468130 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:11:33.468095    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:11:52.266867 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:11:52.266781    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:11:52.280816 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:11:52.280791    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:12:03.469079 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:03.469045    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:12:07.611477 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:07.611440    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"]
Sep 15 17:12:07.611903 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:07.611734    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container" containerID="cri-o://b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b" gracePeriod=30
Sep 15 17:12:07.611903 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:07.611798    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy" containerID="cri-o://684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4" gracePeriod=30
Sep 15 17:12:07.611903 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:07.611798    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-agent" containerID="cri-o://6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38" gracePeriod=30
Sep 15 17:12:07.791324 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:07.791284    2566 generic.go:356] "Generic (PLEG): container finished" podID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerID="684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4" exitCode=2
Sep 15 17:12:07.791480 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:07.791358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerDied","Data":"684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4"}
Sep 15 17:12:08.459469 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:08.459424    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 15 17:12:09.802903 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:09.802871    2566 generic.go:356] "Generic (PLEG): container finished" podID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerID="b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b" exitCode=0
Sep 15 17:12:09.803273 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:09.802927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerDied","Data":"b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b"}
Sep 15 17:12:13.459747 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:13.459707    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 15 17:12:13.465251 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:13.465217    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 15 17:12:18.459480 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:18.459437    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 15 17:12:18.459922 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:18.459613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:12:23.459092 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:23.459046    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 15 17:12:23.464460 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:23.464429    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 15 17:12:28.458855 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:28.458810    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 15 17:12:33.459340 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:33.459294    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.56:8643/healthz\": dial tcp 10.132.0.56:8643: connect: connection refused"
Sep 15 17:12:33.464491 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:33.464463    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.132.0.56:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.132.0.56:8080: connect: connection refused"
Sep 15 17:12:33.464620 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:33.464605    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:12:37.806024 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.805999    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:12:37.872783 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.872730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54lg6\" (UniqueName: \"kubernetes.io/projected/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-api-access-54lg6\") pod \"539d2b25-44f0-4d09-8d67-96f87b0b309e\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") "
Sep 15 17:12:37.872783 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.872782    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-cabundle-cert\") pod \"539d2b25-44f0-4d09-8d67-96f87b0b309e\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") "
Sep 15 17:12:37.872952 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.872821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/539d2b25-44f0-4d09-8d67-96f87b0b309e-kserve-provision-location\") pod \"539d2b25-44f0-4d09-8d67-96f87b0b309e\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") "
Sep 15 17:12:37.872952 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.872841    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls\") pod \"539d2b25-44f0-4d09-8d67-96f87b0b309e\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") "
Sep 15 17:12:37.872952 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.872868    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-rbac-proxy-sar-config\") pod \"539d2b25-44f0-4d09-8d67-96f87b0b309e\" (UID: \"539d2b25-44f0-4d09-8d67-96f87b0b309e\") "
Sep 15 17:12:37.873223 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.873192    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/539d2b25-44f0-4d09-8d67-96f87b0b309e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "539d2b25-44f0-4d09-8d67-96f87b0b309e" (UID: "539d2b25-44f0-4d09-8d67-96f87b0b309e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:12:37.873223 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.873206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "539d2b25-44f0-4d09-8d67-96f87b0b309e" (UID: "539d2b25-44f0-4d09-8d67-96f87b0b309e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:12:37.873405 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.873302    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "539d2b25-44f0-4d09-8d67-96f87b0b309e" (UID: "539d2b25-44f0-4d09-8d67-96f87b0b309e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:12:37.874897 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.874877    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "539d2b25-44f0-4d09-8d67-96f87b0b309e" (UID: "539d2b25-44f0-4d09-8d67-96f87b0b309e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:12:37.874969 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.874929    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-api-access-54lg6" (OuterVolumeSpecName: "kube-api-access-54lg6") pod "539d2b25-44f0-4d09-8d67-96f87b0b309e" (UID: "539d2b25-44f0-4d09-8d67-96f87b0b309e"). InnerVolumeSpecName "kube-api-access-54lg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:12:37.937829 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.937793    2566 generic.go:356] "Generic (PLEG): container finished" podID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerID="6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38" exitCode=137
Sep 15 17:12:37.937960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.937837    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerDied","Data":"6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38"}
Sep 15 17:12:37.937960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.937859    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"
Sep 15 17:12:37.937960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.937940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5" event={"ID":"539d2b25-44f0-4d09-8d67-96f87b0b309e","Type":"ContainerDied","Data":"a78f9cab3d43367604c2d2c6e5a661bce63466334e28ecd1ea77a43cd4a2e812"}
Sep 15 17:12:37.938069 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.937985    2566 scope.go:117] "RemoveContainer" containerID="684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4"
Sep 15 17:12:37.948559 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.948333    2566 scope.go:117] "RemoveContainer" containerID="6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38"
Sep 15 17:12:37.956254 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.956235    2566 scope.go:117] "RemoveContainer" containerID="b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b"
Sep 15 17:12:37.963417 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.963393    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"]
Sep 15 17:12:37.965213 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.964819    2566 scope.go:117] "RemoveContainer" containerID="9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896"
Sep 15 17:12:37.967054 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.967031    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-56686cd547-9jbr5"]
Sep 15 17:12:37.972977 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.972953    2566 scope.go:117] "RemoveContainer" containerID="684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4"
Sep 15 17:12:37.973243 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:12:37.973223    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4\": container with ID starting with 684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4 not found: ID does not exist" containerID="684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4"
Sep 15 17:12:37.973319 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973254    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4"} err="failed to get container status \"684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4\": rpc error: code = NotFound desc = could not find container \"684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4\": container with ID starting with 684ccf6b83f101660ddbff033bc7e4534bb1d44ef041347507247df817042ed4 not found: ID does not exist"
Sep 15 17:12:37.973319 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973279    2566 scope.go:117] "RemoveContainer" containerID="6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38"
Sep 15 17:12:37.973420 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973351    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/539d2b25-44f0-4d09-8d67-96f87b0b309e-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:12:37.973420 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973370    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:12:37.973420 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973381    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54lg6\" (UniqueName: \"kubernetes.io/projected/539d2b25-44f0-4d09-8d67-96f87b0b309e-kube-api-access-54lg6\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:12:37.973420 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973396    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/539d2b25-44f0-4d09-8d67-96f87b0b309e-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:12:37.973420 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973409    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/539d2b25-44f0-4d09-8d67-96f87b0b309e-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:12:37.973693 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:12:37.973502    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38\": container with ID starting with 6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38 not found: ID does not exist" containerID="6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38"
Sep 15 17:12:37.973693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973522    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38"} err="failed to get container status \"6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38\": rpc error: code = NotFound desc = could not find container \"6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38\": container with ID starting with 6c2524b5354e8ceb10a5814aa2ea8459d97ac62a9a3454cb3abf6895e57e1e38 not found: ID does not exist"
Sep 15 17:12:37.973693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973536    2566 scope.go:117] "RemoveContainer" containerID="b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b"
Sep 15 17:12:37.973826 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:12:37.973777    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b\": container with ID starting with b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b not found: ID does not exist" containerID="b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b"
Sep 15 17:12:37.973826 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973793    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b"} err="failed to get container status \"b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b\": rpc error: code = NotFound desc = could not find container \"b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b\": container with ID starting with b8eb28fbc0e8b0260255b29498b94e405b5f9a27a8b0722c0b5d1ae9031a8f7b not found: ID does not exist"
Sep 15 17:12:37.973826 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.973805    2566 scope.go:117] "RemoveContainer" containerID="9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896"
Sep 15 17:12:37.974051 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:12:37.974032    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896\": container with ID starting with 9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896 not found: ID does not exist" containerID="9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896"
Sep 15 17:12:37.974095 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:37.974057    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896"} err="failed to get container status \"9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896\": rpc error: code = NotFound desc = could not find container \"9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896\": container with ID starting with 9f235556fbe7ca7acfb21619b69086d498317234c720263beefb9cd1fc2b5896 not found: ID does not exist"
Sep 15 17:12:38.158868 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:12:38.158834    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" path="/var/lib/kubelet/pods/539d2b25-44f0-4d09-8d67-96f87b0b309e/volumes"
Sep 15 17:13:44.619907 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.619874    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"]
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620183    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="storage-initializer"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620194    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="storage-initializer"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620214    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-agent"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620219    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-agent"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620226    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620232    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620240    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620245    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620292    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-agent"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620298    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kserve-container"
Sep 15 17:13:44.620346 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.620307    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="539d2b25-44f0-4d09-8d67-96f87b0b309e" containerName="kube-rbac-proxy"
Sep 15 17:13:44.623438 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.623421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.626100 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.626071    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:13:44.626100 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.626098    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 15 17:13:44.626304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.626130    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:13:44.626356 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.626332    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 15 17:13:44.626484 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.626465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 15 17:13:44.626623 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.626606    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:13:44.633275 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.633252    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"]
Sep 15 17:13:44.742400 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.742361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.742610 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.742426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.742610 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.742476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gss7\" (UniqueName: \"kubernetes.io/projected/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-api-access-6gss7\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.742610 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.742505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.843026 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.842991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.843213 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.843034    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gss7\" (UniqueName: \"kubernetes.io/projected/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-api-access-6gss7\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.843213 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.843070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.843213 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.843103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.843549 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.843524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.843781 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.843762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.845581 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.845550    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.852259 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.852233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gss7\" (UniqueName: \"kubernetes.io/projected/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-api-access-6gss7\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:44.935678 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:44.935640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:45.209736 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:45.209709    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"]
Sep 15 17:13:45.211706 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:13:45.211679    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74952f4_4ec2_49fe_9e9f_46d59bafc5fd.slice/crio-4387695aca2e9fbe2af2d241bf0fafac07fb0bffa117b87c596df7e5e5045877 WatchSource:0}: Error finding container 4387695aca2e9fbe2af2d241bf0fafac07fb0bffa117b87c596df7e5e5045877: Status 404 returned error can't find the container with id 4387695aca2e9fbe2af2d241bf0fafac07fb0bffa117b87c596df7e5e5045877
Sep 15 17:13:45.237544 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:45.237512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f" event={"ID":"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd","Type":"ContainerStarted","Data":"4387695aca2e9fbe2af2d241bf0fafac07fb0bffa117b87c596df7e5e5045877"}
Sep 15 17:13:46.244052 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:46.244009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f" event={"ID":"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd","Type":"ContainerStarted","Data":"e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65"}
Sep 15 17:13:51.268402 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:51.268323    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd/storage-initializer/0.log"
Sep 15 17:13:51.268402 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:51.268363    2566 generic.go:356] "Generic (PLEG): container finished" podID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" containerID="e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65" exitCode=1
Sep 15 17:13:51.268839 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:51.268441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f" event={"ID":"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd","Type":"ContainerDied","Data":"e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65"}
Sep 15 17:13:52.275581 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:52.275550    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd/storage-initializer/0.log"
Sep 15 17:13:52.276006 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:52.275661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f" event={"ID":"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd","Type":"ContainerStarted","Data":"4d4a0eba47e7235adc6e5b0af5ce1a7f647d8b2800d4fb6d8927103d26f0d8e3"}
Sep 15 17:13:52.438352 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:52.438322    2566 scope.go:117] "RemoveContainer" containerID="ee30ac4e7a4fac99f3558882b583d8167775b476e4f59c87b402690f2774f133"
Sep 15 17:13:52.446411 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:52.446390    2566 scope.go:117] "RemoveContainer" containerID="a373040ee24f6efae8cdceb17490925eed5e9202e88f97267028eb580c4f0380"
Sep 15 17:13:52.453879 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:52.453853    2566 scope.go:117] "RemoveContainer" containerID="82f49f2335fd19a35f46a070e705db93c0ea1962d37090b06f971f264465664e"
Sep 15 17:13:54.285635 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:54.285608    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd/storage-initializer/1.log"
Sep 15 17:13:54.286027 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:54.286010    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd/storage-initializer/0.log"
Sep 15 17:13:54.286076 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:54.286043    2566 generic.go:356] "Generic (PLEG): container finished" podID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" containerID="4d4a0eba47e7235adc6e5b0af5ce1a7f647d8b2800d4fb6d8927103d26f0d8e3" exitCode=1
Sep 15 17:13:54.286148 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:54.286125    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f" event={"ID":"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd","Type":"ContainerDied","Data":"4d4a0eba47e7235adc6e5b0af5ce1a7f647d8b2800d4fb6d8927103d26f0d8e3"}
Sep 15 17:13:54.286225 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:54.286212    2566 scope.go:117] "RemoveContainer" containerID="e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65"
Sep 15 17:13:54.286636 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:54.286611    2566 scope.go:117] "RemoveContainer" containerID="e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65"
Sep 15 17:13:54.297684 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:13:54.297650    2566 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_kserve-ci-e2e-test_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd_0 in pod sandbox 4387695aca2e9fbe2af2d241bf0fafac07fb0bffa117b87c596df7e5e5045877 from index: no such id: 'e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65'" containerID="e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65"
Sep 15 17:13:54.297808 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:13:54.297696    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_kserve-ci-e2e-test(d74952f4-4ec2-49fe-9e9f-46d59bafc5fd)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f" podUID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd"
Sep 15 17:13:54.297808 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:54.297698    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_kserve-ci-e2e-test_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd_0 in pod sandbox 4387695aca2e9fbe2af2d241bf0fafac07fb0bffa117b87c596df7e5e5045877 from index: no such id: 'e9b094f7d8a9daa77f9fa040058affb300d75bafa89c4c2c6cda129c2f6c8a65'"
Sep 15 17:13:54.590297 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:54.590206    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"]
Sep 15 17:13:55.292591 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.292545    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd/storage-initializer/1.log"
Sep 15 17:13:55.423665 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.423643    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd/storage-initializer/1.log"
Sep 15 17:13:55.426990 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.426973    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:55.538011 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.537973    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gss7\" (UniqueName: \"kubernetes.io/projected/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-api-access-6gss7\") pod \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") "
Sep 15 17:13:55.538204 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.538017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-rbac-proxy-sar-config\") pod \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") "
Sep 15 17:13:55.538204 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.538078    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kserve-provision-location\") pod \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") "
Sep 15 17:13:55.538204 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.538114    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-proxy-tls\") pod \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\" (UID: \"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd\") "
Sep 15 17:13:55.538360 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.538338    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" (UID: "d74952f4-4ec2-49fe-9e9f-46d59bafc5fd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:13:55.538429 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.538405    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" (UID: "d74952f4-4ec2-49fe-9e9f-46d59bafc5fd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:13:55.540256 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.540231    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-api-access-6gss7" (OuterVolumeSpecName: "kube-api-access-6gss7") pod "d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" (UID: "d74952f4-4ec2-49fe-9e9f-46d59bafc5fd"). InnerVolumeSpecName "kube-api-access-6gss7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:13:55.540339 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.540324    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" (UID: "d74952f4-4ec2-49fe-9e9f-46d59bafc5fd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:13:55.639470 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.639432    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:13:55.639470 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.639461    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gss7\" (UniqueName: \"kubernetes.io/projected/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-api-access-6gss7\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:13:55.639470 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.639472    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:13:55.639738 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:55.639481    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:13:56.298281 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:56.298252    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f_d74952f4-4ec2-49fe-9e9f-46d59bafc5fd/storage-initializer/1.log"
Sep 15 17:13:56.298707 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:56.298361    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"
Sep 15 17:13:56.298707 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:56.298361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f" event={"ID":"d74952f4-4ec2-49fe-9e9f-46d59bafc5fd","Type":"ContainerDied","Data":"4387695aca2e9fbe2af2d241bf0fafac07fb0bffa117b87c596df7e5e5045877"}
Sep 15 17:13:56.298707 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:56.298431    2566 scope.go:117] "RemoveContainer" containerID="4d4a0eba47e7235adc6e5b0af5ce1a7f647d8b2800d4fb6d8927103d26f0d8e3"
Sep 15 17:13:56.333959 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:56.333933    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"]
Sep 15 17:13:56.342437 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:56.342408    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-69856d5d56-m6t5f"]
Sep 15 17:13:58.159543 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:13:58.159510    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" path="/var/lib/kubelet/pods/d74952f4-4ec2-49fe-9e9f-46d59bafc5fd/volumes"
Sep 15 17:15:26.839240 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.839202    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"]
Sep 15 17:15:26.839725 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.839557    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" containerName="storage-initializer"
Sep 15 17:15:26.839725 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.839580    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" containerName="storage-initializer"
Sep 15 17:15:26.839725 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.839588    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" containerName="storage-initializer"
Sep 15 17:15:26.839725 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.839594    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" containerName="storage-initializer"
Sep 15 17:15:26.839725 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.839657    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" containerName="storage-initializer"
Sep 15 17:15:26.839725 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.839667    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d74952f4-4ec2-49fe-9e9f-46d59bafc5fd" containerName="storage-initializer"
Sep 15 17:15:26.842981 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.842964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.845584 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.845549    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 15 17:15:26.845716 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.845610    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:15:26.845716 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.845661    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:15:26.845716 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.845659    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 15 17:15:26.845896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.845780    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:15:26.845896 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.845817    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 15 17:15:26.853123 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.853090    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"]
Sep 15 17:15:26.862401 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.862380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.862518 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.862411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2txnh\" (UniqueName: \"kubernetes.io/projected/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-api-access-2txnh\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.862518 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.862434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.862627 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.862587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.962966 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.962932    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.963116 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.962977    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2txnh\" (UniqueName: \"kubernetes.io/projected/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-api-access-2txnh\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.963116 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.963009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.963116 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.963082    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.963296 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:15:26.963195    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 15 17:15:26.963296 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:15:26.963287    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls podName:1d1ec8bf-bfc7-48da-9994-3a015a25abb3 nodeName:}" failed. No retries permitted until 2026-09-15 17:15:27.463267041 +0000 UTC m=+2915.831092233 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t" (UID: "1d1ec8bf-bfc7-48da-9994-3a015a25abb3") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Sep 15 17:15:26.963429 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.963406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.963598 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.963562    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:26.973447 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:26.973427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2txnh\" (UniqueName: \"kubernetes.io/projected/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-api-access-2txnh\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:27.466297 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:27.466262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:27.468784 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:27.468758    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:27.754138 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:27.754035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:28.025477 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:28.025395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"]
Sep 15 17:15:28.029707 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:15:28.029679    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1ec8bf_bfc7_48da_9994_3a015a25abb3.slice/crio-dd4472a5e059030f3a7770d7e419a2ade7e97b7a9a2a91b36ce157be5fabe2cd WatchSource:0}: Error finding container dd4472a5e059030f3a7770d7e419a2ade7e97b7a9a2a91b36ce157be5fabe2cd: Status 404 returned error can't find the container with id dd4472a5e059030f3a7770d7e419a2ade7e97b7a9a2a91b36ce157be5fabe2cd
Sep 15 17:15:28.699891 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:28.699855    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t" event={"ID":"1d1ec8bf-bfc7-48da-9994-3a015a25abb3","Type":"ContainerStarted","Data":"cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6"}
Sep 15 17:15:28.700095 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:28.699926    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t" event={"ID":"1d1ec8bf-bfc7-48da-9994-3a015a25abb3","Type":"ContainerStarted","Data":"dd4472a5e059030f3a7770d7e419a2ade7e97b7a9a2a91b36ce157be5fabe2cd"}
Sep 15 17:15:31.714996 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:31.714969    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_1d1ec8bf-bfc7-48da-9994-3a015a25abb3/storage-initializer/0.log"
Sep 15 17:15:31.715340 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:31.715005    2566 generic.go:356] "Generic (PLEG): container finished" podID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" containerID="cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6" exitCode=1
Sep 15 17:15:31.715340 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:31.715069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t" event={"ID":"1d1ec8bf-bfc7-48da-9994-3a015a25abb3","Type":"ContainerDied","Data":"cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6"}
Sep 15 17:15:32.721782 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:32.721751    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_1d1ec8bf-bfc7-48da-9994-3a015a25abb3/storage-initializer/0.log"
Sep 15 17:15:32.722267 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:32.721869    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t" event={"ID":"1d1ec8bf-bfc7-48da-9994-3a015a25abb3","Type":"ContainerStarted","Data":"6c1614fc3637cebdb86d1f6b2b21406ab1ad990ab5b569e82431a5610b33ec04"}
Sep 15 17:15:36.742213 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:36.742187    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_1d1ec8bf-bfc7-48da-9994-3a015a25abb3/storage-initializer/1.log"
Sep 15 17:15:36.742712 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:36.742617    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_1d1ec8bf-bfc7-48da-9994-3a015a25abb3/storage-initializer/0.log"
Sep 15 17:15:36.742712 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:36.742649    2566 generic.go:356] "Generic (PLEG): container finished" podID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" containerID="6c1614fc3637cebdb86d1f6b2b21406ab1ad990ab5b569e82431a5610b33ec04" exitCode=1
Sep 15 17:15:36.742712 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:36.742673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t" event={"ID":"1d1ec8bf-bfc7-48da-9994-3a015a25abb3","Type":"ContainerDied","Data":"6c1614fc3637cebdb86d1f6b2b21406ab1ad990ab5b569e82431a5610b33ec04"}
Sep 15 17:15:36.742824 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:36.742722    2566 scope.go:117] "RemoveContainer" containerID="cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6"
Sep 15 17:15:36.743144 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:36.743122    2566 scope.go:117] "RemoveContainer" containerID="cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6"
Sep 15 17:15:36.754229 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:15:36.754197    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_kserve-ci-e2e-test(1d1ec8bf-bfc7-48da-9994-3a015a25abb3)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t" podUID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3"
Sep 15 17:15:36.754333 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:15:36.754288    2566 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_kserve-ci-e2e-test_1d1ec8bf-bfc7-48da-9994-3a015a25abb3_0 in pod sandbox dd4472a5e059030f3a7770d7e419a2ade7e97b7a9a2a91b36ce157be5fabe2cd from index: no such id: 'cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6'" containerID="cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6"
Sep 15 17:15:36.754333 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:36.754321    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_kserve-ci-e2e-test_1d1ec8bf-bfc7-48da-9994-3a015a25abb3_0 in pod sandbox dd4472a5e059030f3a7770d7e419a2ade7e97b7a9a2a91b36ce157be5fabe2cd from index: no such id: 'cff63a023aa093c4b05be6c1c0408f5f826ecaf84ffba97408cf749afc4422c6'"
Sep 15 17:15:36.885534 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:36.885505    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"]
Sep 15 17:15:37.748183 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.748157    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_1d1ec8bf-bfc7-48da-9994-3a015a25abb3/storage-initializer/1.log"
Sep 15 17:15:37.875764 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.875743    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_1d1ec8bf-bfc7-48da-9994-3a015a25abb3/storage-initializer/1.log"
Sep 15 17:15:37.879125 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.879106    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:37.947661 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.947630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-rbac-proxy-sar-config\") pod \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") "
Sep 15 17:15:37.947827 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.947674    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2txnh\" (UniqueName: \"kubernetes.io/projected/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-api-access-2txnh\") pod \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") "
Sep 15 17:15:37.947827 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.947697    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kserve-provision-location\") pod \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") "
Sep 15 17:15:37.947827 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.947750    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls\") pod \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\" (UID: \"1d1ec8bf-bfc7-48da-9994-3a015a25abb3\") "
Sep 15 17:15:37.948194 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.947976    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1d1ec8bf-bfc7-48da-9994-3a015a25abb3" (UID: "1d1ec8bf-bfc7-48da-9994-3a015a25abb3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:15:37.948194 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.948030    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1d1ec8bf-bfc7-48da-9994-3a015a25abb3" (UID: "1d1ec8bf-bfc7-48da-9994-3a015a25abb3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:15:37.949758 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.949730    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-api-access-2txnh" (OuterVolumeSpecName: "kube-api-access-2txnh") pod "1d1ec8bf-bfc7-48da-9994-3a015a25abb3" (UID: "1d1ec8bf-bfc7-48da-9994-3a015a25abb3"). InnerVolumeSpecName "kube-api-access-2txnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:15:37.949758 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:37.949741    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1d1ec8bf-bfc7-48da-9994-3a015a25abb3" (UID: "1d1ec8bf-bfc7-48da-9994-3a015a25abb3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:15:38.048506 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.048434    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:15:38.048506 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.048469    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:15:38.048506 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.048478    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2txnh\" (UniqueName: \"kubernetes.io/projected/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kube-api-access-2txnh\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:15:38.048506 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.048487    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1d1ec8bf-bfc7-48da-9994-3a015a25abb3-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:15:38.754392 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.754364    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t_1d1ec8bf-bfc7-48da-9994-3a015a25abb3/storage-initializer/1.log"
Sep 15 17:15:38.754862 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.754469    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"
Sep 15 17:15:38.754862 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.754508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t" event={"ID":"1d1ec8bf-bfc7-48da-9994-3a015a25abb3","Type":"ContainerDied","Data":"dd4472a5e059030f3a7770d7e419a2ade7e97b7a9a2a91b36ce157be5fabe2cd"}
Sep 15 17:15:38.754862 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.754609    2566 scope.go:117] "RemoveContainer" containerID="6c1614fc3637cebdb86d1f6b2b21406ab1ad990ab5b569e82431a5610b33ec04"
Sep 15 17:15:38.785313 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.785289    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"]
Sep 15 17:15:38.790774 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:38.790753    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-67ffd4d649-rbj2t"]
Sep 15 17:15:40.158148 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:40.158117    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" path="/var/lib/kubelet/pods/1d1ec8bf-bfc7-48da-9994-3a015a25abb3/volumes"
Sep 15 17:15:41.985474 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.985442    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"]
Sep 15 17:15:41.985886 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.985768    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" containerName="storage-initializer"
Sep 15 17:15:41.985886 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.985780    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" containerName="storage-initializer"
Sep 15 17:15:41.985886 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.985801    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" containerName="storage-initializer"
Sep 15 17:15:41.985886 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.985806    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" containerName="storage-initializer"
Sep 15 17:15:41.985886 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.985852    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" containerName="storage-initializer"
Sep 15 17:15:41.986078 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.985949    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d1ec8bf-bfc7-48da-9994-3a015a25abb3" containerName="storage-initializer"
Sep 15 17:15:41.990342 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.990321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:41.992765 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.992739    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:15:41.992897 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.992779    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Sep 15 17:15:41.993018 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.993002    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:15:41.993104 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.993020    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:15:41.993104 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.993021    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:15:41.993672 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.993657    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Sep 15 17:15:41.993672 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.993667    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:15:41.998758 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:41.998738    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"]
Sep 15 17:15:42.077585 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.077535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.077753 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.077594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.077753 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.077623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.077753 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.077640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxjmt\" (UniqueName: \"kubernetes.io/projected/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-api-access-sxjmt\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.077753 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.077726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.178692 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.178660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.178692 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.178698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.178960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.178735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.178960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.178764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxjmt\" (UniqueName: \"kubernetes.io/projected/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-api-access-sxjmt\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.178960 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.178862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.179173 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.179149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.179547 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.179523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.179661 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.179547    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.181451 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.181427    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.187084 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.187057    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxjmt\" (UniqueName: \"kubernetes.io/projected/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-api-access-sxjmt\") pod \"isvc-pmml-predictor-8674795f59-c29p7\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.302642 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.302553    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:42.580974 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.580950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"]
Sep 15 17:15:42.583297 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:15:42.583260    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff35e7b3_c110_48f9_b2b6_cf1a8f9ebc0e.slice/crio-06b236b7c6f338bc2e693c2593acd8a362acd0db06182cc9d3408eddfb7d66f2 WatchSource:0}: Error finding container 06b236b7c6f338bc2e693c2593acd8a362acd0db06182cc9d3408eddfb7d66f2: Status 404 returned error can't find the container with id 06b236b7c6f338bc2e693c2593acd8a362acd0db06182cc9d3408eddfb7d66f2
Sep 15 17:15:42.776670 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.776626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" event={"ID":"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e","Type":"ContainerStarted","Data":"41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e"}
Sep 15 17:15:42.776842 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:42.776697    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" event={"ID":"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e","Type":"ContainerStarted","Data":"06b236b7c6f338bc2e693c2593acd8a362acd0db06182cc9d3408eddfb7d66f2"}
Sep 15 17:15:46.795072 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:46.795039    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerID="41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e" exitCode=0
Sep 15 17:15:46.795421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:46.795082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" event={"ID":"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e","Type":"ContainerDied","Data":"41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e"}
Sep 15 17:15:53.840019 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:53.839993    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" event={"ID":"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e","Type":"ContainerStarted","Data":"3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997"}
Sep 15 17:15:54.846533 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:54.846488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" event={"ID":"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e","Type":"ContainerStarted","Data":"9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c"}
Sep 15 17:15:54.846915 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:54.846668    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:54.867137 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:54.867090    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podStartSLOduration=2.849220012 podStartE2EDuration="13.867077218s" totalImagesPullingTime="6.9556469s" totalInitContainerRuntime="4.062210306s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:15:54.864792851 +0000 UTC m=+2943.232618050" watchObservedRunningTime="2026-09-15 17:15:54.867077218 +0000 UTC m=+2943.234902416"
Sep 15 17:15:55.851233 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:55.851199    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:15:55.852791 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:55.852765    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:15:56.855724 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:15:56.855681    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:16:01.861901 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:01.861871    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:16:01.862441 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:01.862412    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:16:11.862817 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:11.862777    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:16:21.863018 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:21.862936    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:16:31.862804 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:31.862763    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:16:41.863001 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:41.862960    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:16:51.862934 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:51.862890    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:16:52.293972 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:52.293945    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:16:52.310357 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:16:52.310334    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:17:01.862934 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:01.862896    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.59:8080: connect: connection refused"
Sep 15 17:17:11.863531 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:11.863501    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:17:22.888806 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:22.888757    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"]
Sep 15 17:17:22.889309 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:22.889006    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container" containerID="cri-o://3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997" gracePeriod=30
Sep 15 17:17:22.889309 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:22.889027    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kube-rbac-proxy" containerID="cri-o://9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c" gracePeriod=30
Sep 15 17:17:23.236525 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:23.236440    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerID="9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c" exitCode=2
Sep 15 17:17:23.236673 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:23.236520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" event={"ID":"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e","Type":"ContainerDied","Data":"9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c"}
Sep 15 17:17:26.429100 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.429075    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:17:26.541325 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.541261    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxjmt\" (UniqueName: \"kubernetes.io/projected/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-api-access-sxjmt\") pod \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") "
Sep 15 17:17:26.541325 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.541311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-cabundle-cert\") pod \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") "
Sep 15 17:17:26.541523 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.541343    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-proxy-tls\") pod \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") "
Sep 15 17:17:26.541523 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.541364    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kserve-provision-location\") pod \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") "
Sep 15 17:17:26.541523 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.541384    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-rbac-proxy-sar-config\") pod \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\" (UID: \"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e\") "
Sep 15 17:17:26.541748 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.541722    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" (UID: "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:17:26.541799 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.541759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" (UID: "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:17:26.541836 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.541818    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" (UID: "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:17:26.543388 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.543362    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-api-access-sxjmt" (OuterVolumeSpecName: "kube-api-access-sxjmt") pod "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" (UID: "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e"). InnerVolumeSpecName "kube-api-access-sxjmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:17:26.543483 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.543387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" (UID: "ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:17:26.642500 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.642458    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:17:26.642500 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.642492    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:17:26.642674 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.642508    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxjmt\" (UniqueName: \"kubernetes.io/projected/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-kube-api-access-sxjmt\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:17:26.642674 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.642530    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:17:26.642674 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:26.642540    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:17:27.256778 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.256748    2566 generic.go:356] "Generic (PLEG): container finished" podID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerID="3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997" exitCode=0
Sep 15 17:17:27.256968 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.256805    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"
Sep 15 17:17:27.256968 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.256831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" event={"ID":"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e","Type":"ContainerDied","Data":"3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997"}
Sep 15 17:17:27.256968 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.256890    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7" event={"ID":"ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e","Type":"ContainerDied","Data":"06b236b7c6f338bc2e693c2593acd8a362acd0db06182cc9d3408eddfb7d66f2"}
Sep 15 17:17:27.256968 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.256916    2566 scope.go:117] "RemoveContainer" containerID="9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c"
Sep 15 17:17:27.265635 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.265551    2566 scope.go:117] "RemoveContainer" containerID="3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997"
Sep 15 17:17:27.272816 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.272798    2566 scope.go:117] "RemoveContainer" containerID="41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e"
Sep 15 17:17:27.278048 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.278026    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"]
Sep 15 17:17:27.280456 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.280439    2566 scope.go:117] "RemoveContainer" containerID="9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c"
Sep 15 17:17:27.280772 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:17:27.280741    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c\": container with ID starting with 9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c not found: ID does not exist" containerID="9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c"
Sep 15 17:17:27.280855 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.280771    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c"} err="failed to get container status \"9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c\": rpc error: code = NotFound desc = could not find container \"9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c\": container with ID starting with 9c96389d7e83305e06924b6bcc0e8bf4e7a71d3b6a8a993b0fda0b13ca169e3c not found: ID does not exist"
Sep 15 17:17:27.280855 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.280787    2566 scope.go:117] "RemoveContainer" containerID="3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997"
Sep 15 17:17:27.281037 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:17:27.281017    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997\": container with ID starting with 3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997 not found: ID does not exist" containerID="3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997"
Sep 15 17:17:27.281096 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.281046    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997"} err="failed to get container status \"3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997\": rpc error: code = NotFound desc = could not find container \"3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997\": container with ID starting with 3e89ac187a44c13a4f79f095a4d79e328078146f503d8ea6f8b7516aebc91997 not found: ID does not exist"
Sep 15 17:17:27.281096 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.281062    2566 scope.go:117] "RemoveContainer" containerID="41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e"
Sep 15 17:17:27.281294 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:17:27.281278    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e\": container with ID starting with 41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e not found: ID does not exist" containerID="41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e"
Sep 15 17:17:27.281352 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.281297    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e"} err="failed to get container status \"41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e\": rpc error: code = NotFound desc = could not find container \"41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e\": container with ID starting with 41e0f64f288d29e071713c0bcf038f7260d781ce107a2fb48a69bfb30e73896e not found: ID does not exist"
Sep 15 17:17:27.281998 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:27.281978    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-c29p7"]
Sep 15 17:17:28.159793 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:17:28.159759    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" path="/var/lib/kubelet/pods/ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e/volumes"
Sep 15 17:19:00.620464 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.620415    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"]
Sep 15 17:19:00.620981 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.620937    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="storage-initializer"
Sep 15 17:19:00.620981 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.620958    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="storage-initializer"
Sep 15 17:19:00.620981 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.620969    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kube-rbac-proxy"
Sep 15 17:19:00.620981 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.620978    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kube-rbac-proxy"
Sep 15 17:19:00.621150 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.621001    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container"
Sep 15 17:19:00.621150 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.621010    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container"
Sep 15 17:19:00.621150 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.621099    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kube-rbac-proxy"
Sep 15 17:19:00.621150 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.621110    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff35e7b3-c110-48f9-b2b6-cf1a8f9ebc0e" containerName="kserve-container"
Sep 15 17:19:00.623782 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.623764    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.626353 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.626330    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cdd413-predictor-serving-cert\""
Sep 15 17:19:00.626480 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.626421    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 15 17:19:00.626589 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.626342    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 15 17:19:00.626589 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.626383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-6l9dm\""
Sep 15 17:19:00.626717 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.626396    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-cdd413-kube-rbac-proxy-sar-config\""
Sep 15 17:19:00.626717 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.626334    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 15 17:19:00.626717 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.626430    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 15 17:19:00.633002 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.632978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"]
Sep 15 17:19:00.652154 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.652129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.652265 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.652167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.652265 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.652225    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-cabundle-cert\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.652352 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.652267    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drd59\" (UniqueName: \"kubernetes.io/projected/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-api-access-drd59\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.652352 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.652303    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kserve-provision-location\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.753359 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.753323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.753359 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.753360    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.753614 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.753390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-cabundle-cert\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.753614 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.753412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drd59\" (UniqueName: \"kubernetes.io/projected/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-api-access-drd59\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.753614 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.753439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kserve-provision-location\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.753614 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:19:00.753487    2566 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-primary-cdd413-predictor-serving-cert: secret "isvc-primary-cdd413-predictor-serving-cert" not found
Sep 15 17:19:00.753614 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:19:00.753593    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls podName:dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b nodeName:}" failed. No retries permitted until 2026-09-15 17:19:01.253549771 +0000 UTC m=+3129.621374968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls") pod "isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" (UID: "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b") : secret "isvc-primary-cdd413-predictor-serving-cert" not found
Sep 15 17:19:00.753899 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.753881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kserve-provision-location\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.754062 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.754045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-rbac-proxy-sar-config\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.754101 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.754063    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-cabundle-cert\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:00.761651 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:00.761632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drd59\" (UniqueName: \"kubernetes.io/projected/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-api-access-drd59\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:01.256956 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:01.256907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:01.259329 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:01.259298    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls\") pod \"isvc-primary-cdd413-predictor-b74f8b7dd-2lh44\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") " pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:01.535159 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:01.535067    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:01.808775 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:01.808745    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"]
Sep 15 17:19:01.810240 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:19:01.810212    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8aa434_2c40_4ef8_8cf3_4729b6bcd34b.slice/crio-a3216ef86d24cf7b0a7c27cbec3d29a72b76c7dbc3df439380d59184bafe43b4 WatchSource:0}: Error finding container a3216ef86d24cf7b0a7c27cbec3d29a72b76c7dbc3df439380d59184bafe43b4: Status 404 returned error can't find the container with id a3216ef86d24cf7b0a7c27cbec3d29a72b76c7dbc3df439380d59184bafe43b4
Sep 15 17:19:02.672418 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:02.672377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" event={"ID":"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b","Type":"ContainerStarted","Data":"7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524"}
Sep 15 17:19:02.672642 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:02.672443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" event={"ID":"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b","Type":"ContainerStarted","Data":"a3216ef86d24cf7b0a7c27cbec3d29a72b76c7dbc3df439380d59184bafe43b4"}
Sep 15 17:19:06.692049 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:06.692016    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerID="7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524" exitCode=0
Sep 15 17:19:06.692427 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:06.692071    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" event={"ID":"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b","Type":"ContainerDied","Data":"7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524"}
Sep 15 17:19:07.698873 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:07.698839    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" event={"ID":"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b","Type":"ContainerStarted","Data":"720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf"}
Sep 15 17:19:07.699254 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:07.698900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" event={"ID":"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b","Type":"ContainerStarted","Data":"cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40"}
Sep 15 17:19:07.699254 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:07.699081    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:07.717249 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:07.717205    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podStartSLOduration=3.826594369 podStartE2EDuration="7.717190669s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.8905963s" isStatefulPod=false podCreationTimestamp="2026-09-15 17:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:19:07.715966472 +0000 UTC m=+3136.083791673" watchObservedRunningTime="2026-09-15 17:19:07.717190669 +0000 UTC m=+3136.085015868"
Sep 15 17:19:08.703543 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:08.703512    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:08.704814 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:08.704783    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:19:09.707736 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:09.707689    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:19:14.713175 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:14.713100    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:19:14.713679 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:14.713648    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:19:24.713772 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:24.713729    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:19:34.714334 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:34.714296    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:19:44.713601 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:44.713545    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:19:54.714001 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:19:54.713960    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:20:04.713968 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:20:04.713923    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:20:14.714351 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:20:14.714308    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:20:24.714253 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:20:24.714216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:21:01.857706 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:01.857669    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"]
Sep 15 17:21:01.858205 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:01.857991    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" containerID="cri-o://cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40" gracePeriod=30
Sep 15 17:21:01.858205 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:01.858057    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kube-rbac-proxy" containerID="cri-o://720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf" gracePeriod=30
Sep 15 17:21:02.198195 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:02.198161    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerID="720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf" exitCode=2
Sep 15 17:21:02.198195 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:02.198197    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" event={"ID":"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b","Type":"ContainerDied","Data":"720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf"}
Sep 15 17:21:04.708244 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:04.708200    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.60:8643/healthz\": dial tcp 10.132.0.60:8643: connect: connection refused"
Sep 15 17:21:04.714422 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:04.714386    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.60:8080: connect: connection refused"
Sep 15 17:21:06.100421 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.100399    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:21:06.216471 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.216400    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerID="cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40" exitCode=0
Sep 15 17:21:06.216471 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.216453    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"
Sep 15 17:21:06.216683 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.216458    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" event={"ID":"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b","Type":"ContainerDied","Data":"cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40"}
Sep 15 17:21:06.216683 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.216634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44" event={"ID":"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b","Type":"ContainerDied","Data":"a3216ef86d24cf7b0a7c27cbec3d29a72b76c7dbc3df439380d59184bafe43b4"}
Sep 15 17:21:06.216683 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.216667    2566 scope.go:117] "RemoveContainer" containerID="720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf"
Sep 15 17:21:06.224767 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.224749    2566 scope.go:117] "RemoveContainer" containerID="cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40"
Sep 15 17:21:06.231989 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.231971    2566 scope.go:117] "RemoveContainer" containerID="7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524"
Sep 15 17:21:06.238919 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.238900    2566 scope.go:117] "RemoveContainer" containerID="720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf"
Sep 15 17:21:06.239178 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:21:06.239157    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf\": container with ID starting with 720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf not found: ID does not exist" containerID="720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf"
Sep 15 17:21:06.239235 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.239189    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf"} err="failed to get container status \"720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf\": rpc error: code = NotFound desc = could not find container \"720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf\": container with ID starting with 720a2c4d4c9b66bb44ac6161efa4cbece81eae41817381aad8ac7b188f118ddf not found: ID does not exist"
Sep 15 17:21:06.239235 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.239207    2566 scope.go:117] "RemoveContainer" containerID="cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40"
Sep 15 17:21:06.239460 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:21:06.239442    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40\": container with ID starting with cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40 not found: ID does not exist" containerID="cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40"
Sep 15 17:21:06.239501 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.239468    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40"} err="failed to get container status \"cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40\": rpc error: code = NotFound desc = could not find container \"cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40\": container with ID starting with cc572ce0f5b91812c52d544d0b87ad0071dab07259dbd95377f9b73f4c6cea40 not found: ID does not exist"
Sep 15 17:21:06.239501 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.239485    2566 scope.go:117] "RemoveContainer" containerID="7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524"
Sep 15 17:21:06.239720 ip-10-0-132-85 kubenswrapper[2566]: E0915 17:21:06.239705    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524\": container with ID starting with 7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524 not found: ID does not exist" containerID="7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524"
Sep 15 17:21:06.239767 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.239723    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524"} err="failed to get container status \"7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524\": rpc error: code = NotFound desc = could not find container \"7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524\": container with ID starting with 7c5c1a442a0edb5fc2cf83d0c65f2dd024a651f6940efb379abf98835165f524 not found: ID does not exist"
Sep 15 17:21:06.243026 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.243008    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kserve-provision-location\") pod \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") "
Sep 15 17:21:06.243082 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.243040    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-cabundle-cert\") pod \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") "
Sep 15 17:21:06.243120 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.243095    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-rbac-proxy-sar-config\") pod \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") "
Sep 15 17:21:06.243160 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.243118    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls\") pod \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") "
Sep 15 17:21:06.243160 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.243136    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drd59\" (UniqueName: \"kubernetes.io/projected/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-api-access-drd59\") pod \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\" (UID: \"dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b\") "
Sep 15 17:21:06.243589 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.243321    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" (UID: "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 15 17:21:06.243589 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.243431    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" (UID: "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:21:06.243589 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.243455    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" (UID: "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 15 17:21:06.245237 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.245210    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" (UID: "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 15 17:21:06.245291 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.245264    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-api-access-drd59" (OuterVolumeSpecName: "kube-api-access-drd59") pod "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" (UID: "dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b"). InnerVolumeSpecName "kube-api-access-drd59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 15 17:21:06.344338 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.344309    2566 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:21:06.344338 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.344335    2566 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-proxy-tls\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:21:06.344476 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.344347    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drd59\" (UniqueName: \"kubernetes.io/projected/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kube-api-access-drd59\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:21:06.344476 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.344356    2566 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-kserve-provision-location\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:21:06.344476 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.344364    2566 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b-cabundle-cert\") on node \"ip-10-0-132-85.ec2.internal\" DevicePath \"\""
Sep 15 17:21:06.538283 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.538258    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"]
Sep 15 17:21:06.542043 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:06.542022    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-cdd413-predictor-b74f8b7dd-2lh44"]
Sep 15 17:21:08.158411 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:08.158365    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" path="/var/lib/kubelet/pods/dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b/volumes"
Sep 15 17:21:46.799213 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:46.799163    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zf92w_8e9dce9b-8568-498c-a7f8-cfe8d55410c1/global-pull-secret-syncer/0.log"
Sep 15 17:21:46.928340 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:46.928313    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rhpq7_c4782e77-9ebd-4445-9a7d-59888a8916d4/konnectivity-agent/0.log"
Sep 15 17:21:46.978305 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:46.978279    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-85.ec2.internal_c252aca089984aa62b2a4fbb8eb23028/haproxy/0.log"
Sep 15 17:21:51.145436 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:51.145404    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-lrzv9_0e1009b7-c437-4e0f-801e-96e3106c9524/cluster-monitoring-operator/0.log"
Sep 15 17:21:51.423365 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:51.423293    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjrc8_56262877-35f7-49ee-a216-5d548ec30699/node-exporter/0.log"
Sep 15 17:21:51.448736 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:51.448714    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjrc8_56262877-35f7-49ee-a216-5d548ec30699/kube-rbac-proxy/0.log"
Sep 15 17:21:51.476243 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:51.476225    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjrc8_56262877-35f7-49ee-a216-5d548ec30699/init-textfile/0.log"
Sep 15 17:21:51.909078 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:51.909046    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-7mxvm_5ff9370f-5788-4cad-9d0a-7c324165119f/prometheus-operator-admission-webhook/0.log"
Sep 15 17:21:52.320247 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:52.320156    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:21:52.337794 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:52.337772    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:21:54.056672 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:54.056595    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/2.log"
Sep 15 17:21:54.060852 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:54.060833    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-s5lrt_8f4bb97b-0349-4136-bb5e-2f4c089d367a/console-operator/3.log"
Sep 15 17:21:54.477255 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:54.477229    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-657d8c56dc-d5p7j_117d5a5e-d4a8-4274-9227-e6a687222bd6/console/0.log"
Sep 15 17:21:55.012209 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.012178    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-d7mdp_f90f3bef-0bf8-4c28-8cd1-a464eddbc0f6/volume-data-source-validator/0.log"
Sep 15 17:21:55.347691 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.347596    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"]
Sep 15 17:21:55.348066 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.347931    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="storage-initializer"
Sep 15 17:21:55.348066 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.347943    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="storage-initializer"
Sep 15 17:21:55.348066 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.347950    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kube-rbac-proxy"
Sep 15 17:21:55.348066 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.347957    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kube-rbac-proxy"
Sep 15 17:21:55.348066 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.347969    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container"
Sep 15 17:21:55.348066 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.347976    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container"
Sep 15 17:21:55.348066 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.348030    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kube-rbac-proxy"
Sep 15 17:21:55.348066 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.348041    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd8aa434-2c40-4ef8-8cf3-4729b6bcd34b" containerName="kserve-container"
Sep 15 17:21:55.352186 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.352168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.354945 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.354923    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87zwh\"/\"kube-root-ca.crt\""
Sep 15 17:21:55.355057 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.354926    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-87zwh\"/\"default-dockercfg-klvlx\""
Sep 15 17:21:55.355701 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.355687    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-87zwh\"/\"openshift-service-ca.crt\""
Sep 15 17:21:55.362131 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.362104    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"]
Sep 15 17:21:55.387592 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.387554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm4l4\" (UniqueName: \"kubernetes.io/projected/9e00f6c9-69d4-40d5-acf4-4b236f629d95-kube-api-access-tm4l4\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.387686 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.387606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-podres\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.387686 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.387625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-lib-modules\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.387686 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.387670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-sys\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.387794 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.387688    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-proc\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488563 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm4l4\" (UniqueName: \"kubernetes.io/projected/9e00f6c9-69d4-40d5-acf4-4b236f629d95-kube-api-access-tm4l4\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488599    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-podres\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-lib-modules\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-sys\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488693 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-proc\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488887 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-proc\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488887 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-sys\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488887 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-lib-modules\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.488887 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.488769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9e00f6c9-69d4-40d5-acf4-4b236f629d95-podres\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.496864 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.496844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm4l4\" (UniqueName: \"kubernetes.io/projected/9e00f6c9-69d4-40d5-acf4-4b236f629d95-kube-api-access-tm4l4\") pod \"perf-node-gather-daemonset-bptdp\" (UID: \"9e00f6c9-69d4-40d5-acf4-4b236f629d95\") " pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.662921 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.662888    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:55.931270 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.931244    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"]
Sep 15 17:21:55.933234 ip-10-0-132-85 kubenswrapper[2566]: W0915 17:21:55.933193    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9e00f6c9_69d4_40d5_acf4_4b236f629d95.slice/crio-e1b309bf173a96aedda413cc5e219c63c7314a3764b73c810dc8ca25bfb4d75d WatchSource:0}: Error finding container e1b309bf173a96aedda413cc5e219c63c7314a3764b73c810dc8ca25bfb4d75d: Status 404 returned error can't find the container with id e1b309bf173a96aedda413cc5e219c63c7314a3764b73c810dc8ca25bfb4d75d
Sep 15 17:21:55.934129 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.934104    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9b4d_63e81776-6e13-4425-b7e1-a5f42ea0ce43/dns/0.log"
Sep 15 17:21:55.959095 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:55.959071    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l9b4d_63e81776-6e13-4425-b7e1-a5f42ea0ce43/kube-rbac-proxy/0.log"
Sep 15 17:21:56.100183 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:56.100163    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bpcsp_55b80b8c-5e99-4d98-9193-0dd6056f00e5/dns-node-resolver/0.log"
Sep 15 17:21:56.439654 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:56.439620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp" event={"ID":"9e00f6c9-69d4-40d5-acf4-4b236f629d95","Type":"ContainerStarted","Data":"8f67a85274242204d3226256418227f79f1a3f6f4eeb6710e3cfe7c030f0bd6f"}
Sep 15 17:21:56.440001 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:56.439668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp" event={"ID":"9e00f6c9-69d4-40d5-acf4-4b236f629d95","Type":"ContainerStarted","Data":"e1b309bf173a96aedda413cc5e219c63c7314a3764b73c810dc8ca25bfb4d75d"}
Sep 15 17:21:56.440001 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:56.439758    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:21:56.454091 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:56.454052    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp" podStartSLOduration=1.454039512 podStartE2EDuration="1.454039512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-15 17:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-15 17:21:56.45348572 +0000 UTC m=+3304.821310919" watchObservedRunningTime="2026-09-15 17:21:56.454039512 +0000 UTC m=+3304.821864711"
Sep 15 17:21:56.660683 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:56.660656    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-xvx5n_f64207aa-a010-4064-99ee-1ca2028b239e/node-ca/0.log"
Sep 15 17:21:57.370708 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:57.370626    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7f5d8767b4-qhwn7_6385b954-9257-40b7-a82b-d32c090cd40e/router/0.log"
Sep 15 17:21:57.736682 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:57.736652    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-drgp5_992ed8e3-ebc1-46b8-890f-9150130cf77d/serve-healthcheck-canary/0.log"
Sep 15 17:21:58.065303 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:58.065226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-nhpth_8f85e488-5160-4a79-a4b9-cf5c3c7734b5/insights-operator/0.log"
Sep 15 17:21:58.066133 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:58.066115    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-nhpth_8f85e488-5160-4a79-a4b9-cf5c3c7734b5/insights-operator/1.log"
Sep 15 17:21:58.147395 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:58.147371    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ffhfm_256c5634-bc43-4851-9efb-81dc294aeb42/kube-rbac-proxy/0.log"
Sep 15 17:21:58.166926 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:58.166901    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ffhfm_256c5634-bc43-4851-9efb-81dc294aeb42/exporter/0.log"
Sep 15 17:21:58.185679 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:21:58.185632    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ffhfm_256c5634-bc43-4851-9efb-81dc294aeb42/extractor/0.log"
Sep 15 17:22:00.601536 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:00.601507    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-b589c8cc8-lzm92_d44f95fb-f12a-415d-bd5d-affde1179526/manager/0.log"
Sep 15 17:22:00.655218 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:00.655184    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-25nm7_ec79fddd-81d8-41dc-a6bd-a7f2dd8a2cdd/server/0.log"
Sep 15 17:22:01.168033 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:01.168002    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-2mkmz_353b6b59-1a28-4129-b4ca-4d4f5056b713/seaweedfs/0.log"
Sep 15 17:22:01.191276 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:01.191249    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-wwm86_97ec9d36-dc40-403d-9242-ee2b8ff687fc/seaweedfs-tls-custom/0.log"
Sep 15 17:22:02.454072 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:02.454041    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-87zwh/perf-node-gather-daemonset-bptdp"
Sep 15 17:22:05.568058 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:05.568025    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n8wht_537e42cc-e1d6-4cb1-80d4-fd0209023ec9/kube-storage-version-migrator-operator/1.log"
Sep 15 17:22:05.568920 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:05.568901    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-n8wht_537e42cc-e1d6-4cb1-80d4-fd0209023ec9/kube-storage-version-migrator-operator/0.log"
Sep 15 17:22:06.802127 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:06.802054    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4trk_35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1/kube-multus-additional-cni-plugins/0.log"
Sep 15 17:22:06.823930 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:06.823910    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4trk_35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1/egress-router-binary-copy/0.log"
Sep 15 17:22:06.845404 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:06.845384    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4trk_35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1/cni-plugins/0.log"
Sep 15 17:22:06.865878 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:06.865857    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4trk_35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1/bond-cni-plugin/0.log"
Sep 15 17:22:06.888099 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:06.888080    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4trk_35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1/routeoverride-cni/0.log"
Sep 15 17:22:06.911474 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:06.911458    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4trk_35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1/whereabouts-cni-bincopy/0.log"
Sep 15 17:22:06.932149 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:06.932131    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g4trk_35c5b9ae-2d18-41e2-afdd-a4b6d25b18c1/whereabouts-cni/0.log"
Sep 15 17:22:06.973314 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:06.973288    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-l5jb2_7a28728d-09b1-4b0a-97f4-bd78be16bb3c/kube-multus/0.log"
Sep 15 17:22:07.033223 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:07.033199    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9t8nf_c3f47547-0e34-4911-a0e7-1f136f79d630/network-metrics-daemon/0.log"
Sep 15 17:22:07.051304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:07.051283    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9t8nf_c3f47547-0e34-4911-a0e7-1f136f79d630/kube-rbac-proxy/0.log"
Sep 15 17:22:08.284304 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:08.284279    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc5tl_cdcd828e-624a-4946-9a77-6e65028348a7/ovn-controller/0.log"
Sep 15 17:22:08.304336 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:08.304314    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc5tl_cdcd828e-624a-4946-9a77-6e65028348a7/ovn-acl-logging/0.log"
Sep 15 17:22:08.328254 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:08.328226    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc5tl_cdcd828e-624a-4946-9a77-6e65028348a7/kube-rbac-proxy-node/0.log"
Sep 15 17:22:08.348585 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:08.348554    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc5tl_cdcd828e-624a-4946-9a77-6e65028348a7/kube-rbac-proxy-ovn-metrics/0.log"
Sep 15 17:22:08.367819 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:08.367801    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc5tl_cdcd828e-624a-4946-9a77-6e65028348a7/northd/0.log"
Sep 15 17:22:08.391520 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:08.391499    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc5tl_cdcd828e-624a-4946-9a77-6e65028348a7/nbdb/0.log"
Sep 15 17:22:08.413901 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:08.413880    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc5tl_cdcd828e-624a-4946-9a77-6e65028348a7/sbdb/0.log"
Sep 15 17:22:08.453688 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:08.453666    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-sc5tl_cdcd828e-624a-4946-9a77-6e65028348a7/ovnkube-controller/0.log"
Sep 15 17:22:09.459739 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:09.459708    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-6f8b768455-25zwg_79abacbb-141b-4574-91df-121049c7477e/check-endpoints/0.log"
Sep 15 17:22:09.480043 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:09.480017    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-f9p97_c9072b15-5772-442d-9666-6be71bffbe35/network-check-target-container/0.log"
Sep 15 17:22:10.423337 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:10.423311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-ds64z_405e2425-ed77-4348-b777-be755c35de4d/iptables-alerter/0.log"
Sep 15 17:22:11.058308 ip-10-0-132-85 kubenswrapper[2566]: I0915 17:22:11.058283    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-mcl5w_b5d95411-2800-4973-a18e-d4e8e03c3941/tuned/0.log"